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Preface 


It is no exaggeration to say that linear algebra is a subject of central importance in both mathematics and a 
variety of other disciplines. It is used by virtually all mathematicians and by statisticians, physicists, biologists, 
computer scientists, engineers, and social scientists. Just as the basic idea of first semester differential calculus 
(approximating the graph of a function by its tangent line) provides information about the function, the 
process of linearization often allows difficult problems to be approximated by more manageable linear ones. 
This can provide insight into, and, thanks to ever-more-powerful computers, approximate solutions of the 
original problem. For this reason, people working in all the disciplines referred to above should find the 
Handbook of Linear Algebra an invaluable resource. 

The Handbook is the first resource that presents complete coverage of linear algebra, combinatorial lin- 
ear algebra, and numerical linear algebra, combined with extensive applications to a variety of fields and 
information on software packages for linear algebra in an easy to use handbook format. 


Content 


The Handbook covers the major topics of linear algebra at both the graduate and undergraduate level as well 
as its offshoots (numerical linear algebra and combinatorial linear algebra), its applications, and software 
packages for linear algebra computations. The Handbook takes the reader from the very elementary aspects 
of the subject to the frontiers of current research, and its format (consisting of a number of independent 
chapters each organized in the same standard way) should make this book accessible to readers with divergent 
backgrounds. 


Format 


There are five main parts in this book. The first part (Chapters 1 through Chapter 26) covers linear algebra; 
the second (Chapter 27 through Chapter 36) and third (Chapter 37 through Chapter 49) cover, respectively, 
combinatorial and numerical linear algebra, two important branches of the subject. Applications of linear 
algebra to other disciplines, both inside and outside of mathematics, comprise the fourth part of the book 
(Chapter 50 through Chapter 70). Part five (Chapter 71 through Chapter 77) addresses software packages 
useful for linear algebra computations. 

Each chapter is written by a different author or team of authors, who are experts in the area covered. Each 
chapter is divided into sections, which are organized into the following uniform format: 


* Definitions 
¢ Facts 


* Examples 
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Most relevant definitions appear within the Definitions segment of each chapter, but some terms that 
are used throughout linear algebra are not redefined in each chapter. The Glossary, covering the terminol- 
ogy of linear algebra, combinatorial linear algebra, and numerical linear algebra, is available at the end of 
the book to provide definitions of terms that appear in different chapters. In addition to the definition, 
the Glossary also provides the number of the chapter (and section, thereof) where the term is defined. The 
Notation Index serves the same purpose for symbols. 

The Facts (which elsewhere might be called theorems, lemmas, etc.) are presented in list format, which 
allows the reader to locate desired information quickly. In lieu of proofs, references are provided for all facts. 
The references will also, of course, supply a source of additional information about the subject of the chapter. 
In this spirit, we have encouraged the authors to use texts or survey articles on the subject as references, where 
available. 

The Examples illustrate the definitions and facts. Each section is short enough that it is easy to go back 
and forth between the Definitions/Facts and the Examples to see the illustration of a fact or definition. Some 
sections also contain brief applications following the Examples (major applications are treated in their own 
chapters). 


Feedback 


To see updates and provide feedback and errata reports, please consult the web page for this book: http:// 
www.public.iastate.edu/~lhogben/HLA.html or contact the editor via email, LHogben@iastate.edu, with HLA 
in the subject heading. 
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Preliminaries 


This chapter contains a variety of definitions of terms that are used throughout the rest of the book, but are 
not part of linear algebra and/or do not fit naturally into another chapter. Since these definitions have little 
connection with each other, a different organization is followed; the definitions are (loosely) alphabetized and 
each definition is followed by an example. 


Algebra 


An (associative) algebra is a vector space A over a field F together with a multiplication (x,y) + xy from 
A x Ato A satisfying two distributive properties and associativity, i.e., for alla,b € F andallx,y,z € A: 


(ax + by)z = a(xz) + b(yz), x(ay + bz) = a(xy) + b(xz) (xy)z = x(yz). 


Except in Chapter 69 and Chapter 70 the term algebra means associative algebra. In these two chapters, 
associativity is not assumed. 


Examples: 


The vector space of n x n matrices over a field F with matrix multiplication is an (associative) algebra. 


Boundary 


The boundary 0S of a subset S of the real numbers or the complex numbers is the intersection of the closure 
of S and the closure of the complement of S. 


Examples: 
The boundary of S = {x € C: |z| < l} isoS = {x EC: |z| = 1}. 


Complement 

The complement of the set X in universe S, denoted S \ X, is all elements of S that are not in X. When the 
universe is clear (frequently the universe is {1,...,}) then this can be denoted X°. 

Examples: 


For S = {1,2,3,4,5} and X = {1,3}, S\ X = {2,4,5}. 


Complex Numbers 


Let a,b € R. The symbol i denotes /—1. 


The complex conjugate of a complex number c = a + bi is¢ =a — bi. 
The imaginary part of a + bi is im(a + bi) = b and the real part is re(a + bi) = a. 
The absolute value of c = a + bi is |c| = Va? + b?. 
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The argument of the nonzero complex number re? is @ (withr,6 € Rand0 <rand0 <6 <2z). 
The open right half plane C* is {z € C : re(z) > 0}. 

The closed right half plane Cf is {z € C : re(z) > 0}. 

The open left half plane C~ is {z € C: re(z) < 0}. 

The closed left half plane C~ is {z € C: re(z) < 0}. 


Facts: 
1. |c| = ct 
2. |re?| =r 
3. r’? =rcosO +r sinGi 
4. rel? = rei? 
Examples: 


243i =2-31,14=1.4,1+i = V2e!"/4, 


Conjugate Partition 


Let v = (uj, U2,..., Un) bea sequence of integers such that uj > uw. > --- > uy, > 0. The conjugate partition 
of uv is u* = (uj,..., uF), where uj is the number of js such that u; > i. t is sometimes taken to be 1, but is 
sometimes greater (obtained by extending with 0s). 


Facts: Ift is chosen to be the minimum, and u, > 0, v** = v. 


Examples: 
(4, 3, 2, 2,1)* = (5,4, 2, 1). 


Convexity 
Let V be a real or complex vector space. 

Let {vj, v2,... Vx} € V. A vector of the form a) v) +42v2+- - -+a;V; with all the coefficients a; nonnegative 
and 5° a; = 1 is a convex combination of {v,,v2,... , Vi}. 


A set S C V is convex if any convex combination of vectors in S is in S. 

The convex hull of S is the set of all convex combinations of S and is denoted by Con(S). 

An extreme point of a closed convex set S is a point v € S that is not a nontrivial convex combination of 
other points in S, ie., ax + (1 — a)y = vand0 <a < 1impliesx = y =v. 

A convex polytope is the convex hull of a finite set of vectors in R”. 

Let S C V be convex. A function f:S — R is convex if for alla e RJ0<a<1,xyeS, flax+(1— 


a)jy) < af(x) + (1 —a) f(y). 


Facts: 


1. Aset S C V is convex if and only if Con(S) = S. 
2. The extreme points of Con(S) are contained in S. 
3. [HJ85] Krein-Milman Theorem: A compact convex set is the convex hull of its extreme points. 


Examples: 
1. [1.9,0.8]7 is a convex combination of [1,—1]7 and [2,1]7, since [1.9,0.8]? = 0.1[1,-1]7 + 
0.9[2, 1]7. 
2. The set K ofall v € R® such that v; > 0,i = 1,2,3 is a convex set. Its only extreme point is the 
zero vector. 
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Elementary Symmetric Function 


The kth elementary symmetric function of a;,i = 1,...,n is 


Sk(Q1, 6. -5An) = Oj, Aj, ...Qj,. 


1 <i) <in<-+'<ip<n 


Examples: 


S2(Q1, 2, 013) = at0r2 + 0103 + 2013, 
S1 (045.265 Qn) = Oy $2 +++ + Qq, Sy (Q4,...50y) = 02... Ap. 


Equivalence Relation 


A binary relation = in a nonempty set S is an equivalence relation if it satisfies the following conditions: 


1. (Reflexive) For alla € S,a =a. 
2. (Symmetric) For alla, b € S,a = b implies b =a. 
3. (Transitive) For alla,b,c € S,a =banda = b implya =c. 


Examples: 


Congruence mod n is an equivalence relation on the integers. 


Field 


A field is a set F with at least two elements together with a function F x F — F called addition, denoted 
(a,b) > a+b, anda function F x F > F called multiplication, denoted (a,b) — ab, which satisfy the 
following axioms: 


Commutativity) For eacha,b € F,a+b=b+aandab = ba. 

Associativity) For each a,b,c € F,(a+b)+c =a+(b+c) and (ab)c = a(bc). 

Identities) There exist two elements 0 and 1 in F such thatO0 +a =aand1la =a foreacha € F. 

Inverses) For each a € F, there exists an element —a € F such that (—a) + a = O. For each 
nonzero a € F, there exists an element a~! € F such that a~!a = 1. 

5. (Distributivity) For each a,b,c € F,a(b+c)=ab+ac. 


tee a 
aa 


Examples: 


The real numbers, R, the complex numbers, C, and the rational numbers, Q, are all fields. The set of integers, 
Z, is not a field. 


Greatest Integer Function 


The greatest integer or floor function |x| (defined on the real numbers) is the greatest integer less than or 
equal to x. 


Examples: 
[1.5] =1, [1] = 1, [—1.5] =—2. 
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Group 


(See also Chapter 67 and Chapter 68.) 
A group is a nonempty set G with a function G x G — G denoted (a,b) — ab, which satisfies the following 
axioms: 


1. (Associativity) For each a,b,c € G, (ab)c = a(bc). 
2. (Identity) There exists an element e € G such that ea = a = ae foreacha € G. 
3. (Inverses) For each a € G, there exists an element a~! € G such that a~!a = e = aa™!. 


A group is abelian if ab = ba for alla,b € G. 


Examples: 


1. Any vector space is an abelian group under +. 
2. The set of invertible n x n real matrices is a group under matrix multiplication. 
3. The set of all permutations of a set is a group under composition. 


Interlaces 


Let a; > a) > +--+ > a, and by > by > --- > b,_1, two sequences of real numbers arranged in decreasing 
order. Then the sequence {b;} interlaces the sequence {a;} if a, < Dy—) < ay—1--- < by < a). Further, if 
all of the above inequalities can be taken to be strict, the sequence {b;} strictly interlaces the sequence {a;}. 
Analogous definitions are given when the numbers are in increasing order. 


Examples: 


7 > 2.2 > —1 strictly interlaces 11 > 7 > 0 > —2.6. 


Majorization 
Let @ = (41,42,... 54), B = (b), b2,... , by) be sequences of real numbers. 
alt= (alas, Peer , a) isthe permutation ofa with entries in nonincreasing order, ie,ay > ay Se ay. 
at= (a}, at, bias a}) is the permutation of @ with entries in nondecreasing order, Le., a} < at <...< 
t 
a’, 


n 
a weakly majorizes 6, writtena >, 6B or B Xy a, if: 


k k 
Soa} = Sob} forallk=1,...n. 
i=1 i=l 


a majorizes B, writtena > B or B < a, ifa >w Band >i, a; = 0j_, Ui. 


Examples: 


1. Ifa = (2,2, —1.3,8,7.7), then a’ = (8, 7.7, 2,2, —1.3) anda? = (—1.3,2,2,7.7,8). 
2. (5,3,1.5,1.5,1) > (4,3,2,2,1) and (6,5,0) >, (4,3,2). 


Metric 


A metric on a set S is a real-valued function f:S x S — R satisfying the following conditions: 
1. Forallx,y € S, f(x,y) > 0. 
2. Forallx € S, f(x,x) = 0. 
3. Forallx,y € S, f(x,y) = 0 implies x = y. 
4. Forallx,y € S, f(x,y) = f(y). 
5. Forallx,y,z € S, f(x,y) + fly,z) = f(x,z). 


A metric is intended as a measure of distance between elements of the set. 
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Examples: 


If || - || isa norm on a vector space, then f(x, y) = ||x — y|| is a metric. 


Multiset 


A multiset is an unordered list of elements that allows repetition. 


Examples: 


Any set is a multiset, but {1, 1,3, —2, —2, —2} is a multiset that is not a set. 


O and o 


Let, f, g be real valued functions of N or R,iie., f6g:N— Ror f,g:R-R. 
f is O(g) (big-oh of g) if there exist constants C, k such that | f(x)| < C|g(x)| for all x > k. 


f is o(g) (little-oh of g) if limy_ +50 | fio| = 0, 


Examples: 


x? + x is O(x?) and In x is o(x). 


Path-connected 


A subset S of the complex numbers is path-connected if for any x, y € S there exists a continuous function 
p: [0,1] > S with p(0) = x and p(1) = y. 


Examples: 
S={zeC:1 < |z| < 2} and the line {a + bi : a = 2b + 3} are path-connected. 


Permutations 


A permutation is a one-to-one onto function from a set to itself. 


The set of permutations of {1,...,} is denoted S,,. The identity permutation is denoted ¢,,. In this book, 
permutations are generally assumed to be elements of S,, for some n. 
A cycle or k-cycle is a permutation t such that there is a subset {a,...,a,} of {1,...,n} satisfying t(a;) = 


aj4, and t(ax) = aj; this is denoted t = (a), 42,...,a,). The length of this cycle is k. 

A transposition is a 2-cycle. 

A permutation is even (respectively, odd) if it can be written as the product of an even (odd) number of 
transpositions. 

The sign of a permutation t, denoted sgn T, is +1 if t is even and —1 if t is odd. 

Note: Permutations are functions and act from the left (see Examples). 


Facts: 


1. Every permutation can be expressed as a product of disjoint cycles. This expression is unique up to 
the order of the cycles in the decomposition and cyclic permutation within a cycle. 

2. Every permutation can be written as a product of transpositions. If some such expression 
includes an even number of transpositions, then every such expression includes an even num- 
ber of transpositions. 

3. S, with the operation of composition is a group. 
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Examples: 
1. Ift = (1523) € So, then t(1) = 5, t(2) = 3, t(3) = 1,7 (4) = 4,7 (5) = 2,7(6) = 6. 
2. (123)(12)=(13). 
3. sgn(1234) = —1, because (1234) = (14)(13)(12). 


Ring 


(See also Section 23.1) 
A ring is a set R together with a function R x R > R called addition, denoted (a,b) > a+b, anda function 
Rx R— R called multiplication, denoted (a,b) — ab, which satisfy the following axioms: 


1. (Commutativity of +) For eacha,b € R,a+b=b-+a. 

2. (Associativity) For each a,b,c € R,(a+b)+c=a+(b+c)and (ab)c = a(bc). 
3. (+ identity) There exists an element 0 in R such thatO0 +a =a. 

4. (+ inverse) For eacha € R, there exists an element —a € R such that (—a) +a =0. 
5. (Distributivity) For each a,b,c € R,a(b+c) =ab-+acand(a+b)c =ac+be. 


— 


A zero divisor in a ring R is a nonzero element a € R such that there exists a nonzero b € R withab = 0 
or ba = 0. 


Examples: 


* The set of integers, Z, is a ring. 
* Any field is a ring. 
* Let F bea field. Then F”*”, with matrix addition and matrix multiplication as the operations, is 


F 1 0 0 0 nce ‘ 
aring. Ey, = E and Ey = } j are zero divisors since E,,E 7 = 0». 


Sign 


(For sign of a permutation, see permutation.) 
The sign of a complex number is defined by: 


sign(z) = ae | ia s 


If z is a real number, this sign function yields 1 or —1. 
This sign function is used in numerical linear algebra. 
The sign of a real number (as used in sign patterns) is defined by: 


+, ifa>0; 
sgn(a)= ¢ 0, ifa=0; 
—, ifa <0. 


This sign function is used in combinatorial linear algebra, and the product of a sign and a real number is 
interpreted in the obvious way as a real number. 
Warning: The two sign functions disagree on the sign of 0. 
Examples: 
sgn(—1.3) =—, sign(—1.3) = —1, sgn(0) = 0, sign(0) = 1, 
(1 +1) 
V2 


sign(1 + i)= 
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Throughout this chapter, F will denote a field. The references [Lay03], [Leo02], and [SIF00] are good 
sources for more detail about much of the material in this chapter. They discuss primarily the field of real 
numbers, but the proofs are usually valid for any field. 


1.1 Vector Spaces 


Vectors are used in many applications. They often represent quantities that have both direction and 
magnitude, such as velocity or position, and can appear as functions, as n-tuples of scalars, or in other 
disguises. Whenever objects can be added and multiplied by scalars, they may be elements of some vector 
space. In this section, we formulate a general definition of vector space and establish its basic properties. 
An element of a field, such as the real numbers or the complex numbers, is called a scalar to distinguish it 
from a vector. 


Definitions: 


A vector space over F is a set V together with a function V x V — V called addition, denoted (x,y) > 
x+y, and a function F x V — V called scalar multiplication and denoted (c,x) — cx, which satisfy 
the following axioms: 


Commutativity) For each x, y€ V,x +y=y+x. 

Associativity) For each x, y,z € V, (x+y) +z=x+(y+z). 

Additive identity) There exists a zero vector in V, denoted 0, such that 0 + x = x for each x € V. 
Additive inverse) For each x € V, there exists —x € V such that (—x) +x=0. 

Distributivity) For eacha € F andx,y € V,a(x+y) =ax-+ ay. 

Distributivity) For each a, b € F andxeé V, (a+b) x=ax-+ bx. 
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7. (Associativity) For each a,b € F andxe V, (ab) x= a(bx). 
8. For eachx € V, Ix=x. 


The properties that for allx, ye V,anda € F,x+ye V andaxe V, are called closure under addition 
and closure under scalar multiplication, respectively. The elements of a vector space V are called vectors. 
A vector space is called real if F = R, complex if F = C. 

If n is a positive integer, F” denotes the set of all ordered n-tuples (written as columns). These are 


x] yal 
sometimes written instead asrows[x, +++ X,]or(x1,...,%X,).Forx= | : |,y= |: | €F"andc ¢€ F, 
lal ba 
x+y CX, 
define addition and scalar multiplication coordinate-wise: x + y = : andcx= | : |. Let 0 
Xn + Yn CXy 


denote the n-tuple of zeros. For x € F", x; is called the j" coordinate of x. 

A subspace of vector space V over field F is a subset of V, which is itself a vector space over F when 
the addition and scalar multiplication of V are used. If S; and S) are subsets of vector space V, define 
Si +S. ={x+y:xe S,andye S)}. 


Facts: 


Let V be a vector space over F. 


1. F” is a vector space over F. 
2. [FISO3, pp. 11-12] (Basic properties of a vector space): 
* The vector 0 is the only additive identity in V. 
* For each x € V, —x is the only additive inverse for x in V. 
* For eachx € V, —x= (—1)x. 
° Ifa € F andxe V, then ax = 0 ifand only ifa = 0orx=0. 
* (Cancellation) Ifx, y,ze Vandx+y=x+z, theny =z. 
3. [FISO3, pp. 16-17] Let W be a subset of V. The following are equivalent: 
* W isa subspace of V. 
* W is nonempty and closed under addition and scalar multiplication. 
* 0¢ Wand for any x,y € Wanda,b € F,ax+ bye W. 
4. For any vector space V, {0} and V itself are subspaces of V. 
[FISO3, p. 19] The intersection of any nonempty collection of subspaces of V is a subspace of V. 
6. [FISO3, p. 22] Let W, and W, be subspaces of V. Then W, + W) is a subspace of V containing W, 
and W). It is the smallest subspace that contains them in the sense that any subspace that contains 
both W, and W; must contain W; + W. 
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Examples: 
1. The set R” of all ordered n-tuples of real numbers is a vector space over R, and the set C” of 
3 
all ordered n-tuples of complex numbers is a vector space over C. For instance, x = 0} and 
—1 
21 342i —2i —2 
y= 4 are elements of C?;x + y= 4 |,-y= —4 |,andiy= 4i 
2—31 1-31 —24+ 31 34 21 


2. Notice R" is a subset of C” but not a subspace of C’, since R" is not closed under multiplication 
by nonreal numbers. 


Vectors, Matrices, and Systems of Linear Equations 1-3 


3. The vector spaces R, R’, and R’ are the usual Euclidean spaces of analytic geometry. There are 
three types of subspaces of R’: {0}, a line through the origin, and R’ itself. There are four types of 
subspaces of R’: {0}, aline through the origin, a plane through the origin, and R’ itself. For instance, 
let v= (5, —1,—1) and w= (0,3, —2). The lines W, = {sv:s € R} and W, = {sw:s € R} are 
subspaces of R®. The subspace W; + Wy = {sv-+tw:s,t € R} isa plane. The set {sv-+ w:s € R} 
is a line parallel to W,, but is not a subspace. (For more information on geometry, see Chapter 65.) 

4, Let F [x] be the set of all polynomials in the single variable x, with coefficients from F. To add 
polynomials, add coefficients of like powers; to multiply a polynomial by an element of F, multi- 
ply each coefficient by that scalar. With these operations, F [x] is a vector space over F. The zero 
polynomial z, with all coefficients 0, is the additive identity of F [x]. For f € F[x], the function 
—f defined by — f(x) = (—1) f (x) is the additive inverse of f. 

5. In F[x], the constant polynomials have degree 0. For n > 0, the polynomials with highest power 
term x" are said to have degree n. For a nonnegative integer n, let F[x;n] be the subset of F [x] 
consisting of all polynomials of degree n or less. Then F [x; 1] is a subspace of F [x]. 

6. When n > 0, the set of all polynomials of degree exactly n is not a subspace of F [x] because it is 
not closed under addition or scalar multiplication. The set of all polynomials in R[x] with rational 
coefficients is not a subspace of R[x] because it is not closed under scalar multiplication. 

7. Let V be the set of all infinite sequences (a), a2, 43,...), where each aj € F. Define addition and 
scalar multiplication coordinate-wise. Then V is a vector space over F. 

8. Let X beanonempty set and let F(X, F) be the set of all functions f: X > F.Let fog € F(X, F) 
and define f + g and cf pointwise, as (f + g)(x) = f(x) + g(x) and (cf) (x) = cf (x) forall x € X. 
With these operations, F(X, F) is a vector space over F. The zero function is the additive identity 
and (—1) f = —f, the additive inverse of f. 

9. Let X beanonempty subset of R". The set C(X) ofall continuous functions f: X + Risasubspace 
of F(X, R). The set D(X) ofall differentiable functions f: X > R is a subspace of C(X) and also 
of F(X,R). 


1.2 Matrices 


Matrices are rectangular arrays of scalars that are used in a great variety of ways, such as to solve linear 
systems, model linear behavior, and approximate nonlinear behavior. They are standard tools in almost 
every discipline, from sociology to physics and engineering. 


Definitions: 
ay. eee a Ip 


Anm x p matrix over F isan m x p rectangular array A = : : | with entries from F. The 


ins. 24% inp 
notation A = [a;,j] that displays a typical entry is also used. The element a; of the matrix A is called the (i, j) 
entry of A and can also be denoted (A),. The shape (or size) of A is m x p, and A is square if m = p; in 
this case, m is also called the size of A. Two matrices A = [aj] and B = [bjj] are said to be equal if they have 
the same shape and aj; = bj for alli, j. Let A = [aj] and B = [b,j] be m x p matrices, and let c bea scalar. 
Define addition and scalar multiplication on the set of all m x p matrices over F entrywise, as A+ B = 
[aj + bj] and cA = [cajj]. The set of all m x p matrices over F with these operations is denoted F”™?. 


a lj 

If Aism x p,rowiis[aj, ..., ap] and column j is | : |. These are called a row vector and 
Amy 

a column vector respectively, and they belong to F"*! and F!*", respectively. The elements of F" are 


identified with the elements of F"*! (or sometimes with the elements of F!*”). Let O.np denote the m x p 
matrix of zeros, often shortened to 0 when the size is clear. Define —A = (—1)A. 
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by 
Let A=[a; ... ap] ¢ F”™*?, where a; is the jth column of A, and letb = | : | € F?*!. The 
bp 
matrix-vector product of A and b is Ab = ba; +--+ + bpap. Notice Ab is m x 1. 
If A € F™P andC = [cy ... c,] € F?*", define the matrix product of A and C as AC = 


[Ac; ...  Ac,]. Notice AC ism x n. 

Square matrices A and B commute if AB = BA. When i = j, aj is a diagonal entry of A and the set of 
all its diagonal entries is the main diagonal of A. Wheni ¥ j, a; is an off-diagonal entry. 

The trace of A is the sum of all the diagonal entries of A, tr A = san Aii- 

A matrix A = [a,j] is diagonal if aj; = 0 whenever i 4 j, lower triangular if aj; = 0 whenever i < j, 
and upper triangular if aj; = 0 whenever i > j. A unit triangular matrix is a lower or upper triangular 
matrix in which each diagonal entry is 1. 

The identity matrix [,,, often shortened to I when the size is clear, is the n x m matrix with main 
diagonal entries 1 and other entries 0. 

A scalar matrix is a scalar multiple of the identity matrix. 

A permutation matrix is one whose rows are some rearrangement of the rows of an identity matrix. 

Let A € F"*?. The transpose of A, denoted A’, is the p X m matrix whose (i, j) entry is the (j,7) 
entry of A. 

The square matrix A is symmetric if A’ = A and skew-symmetric if A’ = — A. 

When F = C, thatis, when A has complex entries, the Hermitian adjoint of A is its conjugate transpose, 
A* = A!; that is, the (i, j) entry of A* is aj. Some authors, such as [Leo02], write A® instead of A*. 

The square matrix A is Hermitian if A* = A and skew-Hermitian if A* = — A. 

Let a be a nonempty set of row indices and £ a nonempty set of column indices. A submatrix of A is 
a matrix A[a, 6] obtained by choosing the entries of A, which lie in rows w and columns f. A principal 
submatrix of A is a submatrix of the form A[q, a]. A leading principal submatrix of A is one of the form 
A[{1,...,k}, (1.2.5 kK}. 


Facts: 

1. [SIFOO, p.5] F”*? is a vector space over F. That is, if0, A, B,C € F™*?, andc,d € F, then: 
*A+B=B+A 
*(A+B)+C=A+(B+C) 
* A+0=0+A=A 
* A+(—A)=(-—A)+A=0 
* c(A+ B)=cA+cB 
*(c+d)A=cA+dA 
* (cd) A=c(dA) 
*1A=A 

2. If Ae F™*? and C € F?*", the (i, j) entry of AC is (AC) = pyar ajxaxj. This is the matrix 
product of rowi of A and column j of C. 


3. [SIFOO, p. 88] Let c € F, let A and B be matrices over F, let I denote an identity matrix, and 
assume the shapes allow the following sums and products to be calculated. Then: 


* AI=IA=A 

« A0=Oand0A=0 

A(BC) = (AB)C 

A(B + C) = AB+ AC 

* (A+ B)C =AC+ BC 

c(AB) = A(cB) = (cA)B for any scalar c 
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4. [SIFOO, p. 5 and p. 20] Letc € F, let A and B be matrices over F, and assume the shapes allow the 
following sums and products to be calculated. Then: 


- CASS 
* (A+B)? = AT + BT 
* (cA)? =cA™ 
° (AB)’ =Br at 
5. [Leo02, pp. 321-323] Let c € C, let A and B be matrices over C, and assume the shapes allow the 
following sums and products to be calculated. Then: 


. (A*)*#=A 
° (A+ B)* = A* + B* 
* (cA)* =¢A* 


* (AB)* = B* A* 
6. If Aand B aren x n and upper (lower) triangular, then AB is upper (lower) triangular. 


Examples: 
7 
1 2 3 oe 1 2 3 —4 
1. vta=[j 5 ;] and = = By definition, ab= 7 [{] +-8 [2] -9 [2] = [7 tana 
: : : 1-74+2-8-—3.-9 —4 
calculation of Ab can be done more quickly using Fact 2: Ab = pees! = ek 
1 -l 8 
2. Let A = E = 3-8 = E 4 sand c= 1 3  O|. Then A+B = 
2 0 8 1 2 -5 
1 2 —2 
—2 5 4 2 6 8 ‘ 
| 3 2 | and 2A = i 0 | . The matrices A + C, BA, and AB are not defined, but 
1 —l 8 
2 —2 0 


2 —2 
3. Even when the shapes of A and B allow both AB and BA to be calculated, AB and BA are not usually 
; 1 0 a b a b a 2b 
equal. For instance, let A = F | and B = k il ; then AB= E a and BA= ; 7 : 


which will be equal only if b = c = 0. 
4. The product of matrices can be a zero matrix even if neither has any zero entries. For example, if 


A= E | and B = f 1 , then AB = k 4 . Notice that BA is also defined but has no 


2 —2 1 
; 3° =3 
tries: BA = : 
zero entries | a 4 
; aes iy 1 0 . Ls u 1 0 0 
5. The matrices |0 0 0} and cee are diagonal, }2 0 0} and — are 
0 0 -9 T5359 
1 —4 7 ; ; : 1 0 0 
lowertriangular,and |0 1 2) and ae are upper triangular. Thematrix }2 1 0 
0 0 —9 00 0 15 1 


is unit lower triangular, and its transpose is unit upper triangular. 
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10. 


11. 


12, 
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001 001 
Examples of permutation matrices include every identity matrix, 1 i 0 1 Oj,and]1 0 0 
100 010 
; 1+i 142i 1-i 1-2: 
1 —31 4 : : 
Let A = i aE = | .Then A? = | —3i 51 and A* = | 33 —5i 
4 0 4 0 
1 2 3 i 2 3421 
. The matrices |2 4 5] and 2 4-1 51 are symmetric. 
3 5 6 3421 51 6 
0 2 3 0 2 34+21 
. The matrices }—2 0 5} and —2 0 —5 | are skew-symmetric. 
—3 -5 0 —3-21 5 0 
1 2+1 1-31 
The matrix | 2—i 0 1 is Hermitian, and any real symmetric matrix, such as 
14+ 31 1 6 
4 2 3 
2 0 5], is also Hermitian. 
3 5 -l1 


i 2 -—3+4+21 
The matrix —2 4i 5 is skew-Hermitian, and any real skew-symmetric matrix, 
3+2i1 —5 0 


0 2 —-3 
such as | —2 0 5] ,is also skew-Hermitian. 
3. -5 0 
1 2 3 4 3 
Let A = ae ae ahs RowlofAis[1 2 3 4], column3 is ‘ and the submatrix 
= 9 10 11 12] ° 5 ll]? 
13. 14 #415 «#16 15 
2 3 4 
in rows {1, 2, 4} and columns {2, 3, 4} is A[{1, 2,4}, {2,3,4}] = | 6 7 8] .A principal 
14 15 16 
1 2 4 
submatrix of A is A[{1, 2, 4}, {1,2,4}] = | 5 6 8]. The leading principal submatrices of 
13 14 «+16 
> 1 2 3 
Aare [1], P l 5 6 7|,and A itself. 


9 10 11 
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1.3. Gaussian and Gauss--Jordan Elimination 


Definitions: 


Let A be a matrix with m rows. 
When a row of A is not zero, its first nonzero entry is the leading entry of the row. The matrix A is in 
row echelon form (REF) when the following two conditions are met: 


1. Any zero rows are below all nonzero rows. 
2. For each nonzero rowi,i < m-— 1, either rowi + 1 is zero or the leading entry of rowi + 1isina 
column to the right of the column of the leading entry in row 7. 


The matrix A is in reduced row echelon form (RREF) if it is in row echelon form and the following 
third condition is also met: 


3. If ajx is the leading entry in row i, then aj, = 1, and every entry of column k other than ajz is zero. 
Elementary row operations on a matrix are operations of the following types: 


1. Add a multiple of one row to a different row. 
2. Exchange two different rows. 
3. Multiply one row by a nonzero scalar. 


The matrix A is row equivalent to the matrix B if there is a sequence of elementary row operations 
that transforms A into B. The reduced row echelon form of A, RREF(A), is the matrix in reduced row 
echelon form that is row equivalent to A. A row echelon form of A is any matrix in row echelon form 
that is row equivalent to A. The rank of A, denoted rank A or rank(A), is the number of leading entries 
in RREF(A). If A is in row echelon form, the positions of the leading entries in its nonzero rows are called 
pivot positions and the entries in those positions are called pivots. A column (row) that contains a pivot 
position is a pivot column (pivot row). 

Gaussian Elimination is a process that uses elementary row operations in a particular way to change, 
or reduce, a matrix to row echelon form. Gauss—Jordan Elimination is a process that uses elementary row 
operations in a particular way to reduce a matrix to RREF. See Algorithm 1 below. 


Facts: 
Let Ac F™*?, 


[Lay03, p. 15] The reduced row echelon form of A, RREF(A), exists and is unique. 

A matrix in REF or RREF is upper triangular. 

Every elementary row operation is reversible by an elementary row operation of the same type. 
If A is row equivalent to B, then B is row equivalent to A. 

If A is row equivalent to B, then RREF(A) = RREF(B) and rank A = rank B. 

The number of nonzero rows in any row echelon form of A equals rank A. 


STO tS 


If B is any row echelon form of A, the positions of the leading entries in B are the same as the 
positions of the leading entries of RREF(A). 

8. [Lay03, pp. 17-20] (Gaussian and Gauss—Jordan Elimination Algorithms) When one or more piv- 
ots are relatively small, using the algorithms below in floating point arithmetic can yield inaccurate 
results. (See Chapter 38 for more accurate variations of them, and Chapter 75 for information on 
professional software implementations of such variations.) 
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Algorithm 1. Gaussian and Gauss-Jordan Elimination 


Let A € F’”*?. Steps 1 to 4 below do Gaussian Elimination, reducing A to a matrix that is in row 
echelon form. Steps 1 to 6 do Gauss—Jordan Elimination, reducing A to RREF(A). 


1. Let U = Aandr = 1. If U =0, U is in RREF. 

2. If U £0, search the submatrix of U in rows r to m to find its first nonzero column, k, and 
the first nonzero entry, ajx, in this column. Ifi > 1, exchange rows r and i in U, thus getting a 
nonzero entry in position (r,k). Let U be the matrix created by this row exchange. 

3. Add multiples of row r to the rows below it, to create zeros in column k below row r. Let U 
denote the new matrix. 

4. Ifeitherr = m— 1 orrowsr +1,...,mareall zero, U is now in REF. Otherwise, letr = r + 1 
and repeat steps 2, 3, and 4. 

5. Let ki,...,k, be the pivot columns of U, so (1,k;),...,(s,k,) are the pivot positions. Fori = s, 
s—1,...,2,add multiples of row to the rows above it to create zeros in column k; above rowi. 

6. Fori = 1,...,5s, divide rows by its leading entry. The resulting matrix is RREF(A). 


Examples: 
1. The RREF of a zero matrix is itself, and its rank is zero. 


1 3 4 —8 1 3 4 —8 
2, LettA=}]0 0 2 4)/andB=J0 0 O_ 4{. Both are upper triangular, but A is in REF 
0 0 0 0 0 0 1 0 


and B is not. Use Gauss—Jordan Elimination to calculate RREF(A) and RREF(B). 


1 3 0 —-16 
For A, add (—2)(row2) torow1 and multiply row2 by $. This yields RREF(A)= ]0 0 1 2 
0 0 0 0 
13 4 -8 
For B, exchange rows 2 and3toget |0 0 1 0}, which is in REF. Then add 2(row 3) to 
0 0 0 4 


row | to get a new matrix. In this new matrix, add (—4) (row 2) to row 1, and multiply row 3 by i 


13 0 0 
This yields RREF(B) = |]0 0 1 O}. 
0 0 0 1 


Observe that rank (A) = 2 and rank (B) = 3. 


3. Apply Gauss—Jordan Elimination to A = 


Fe CO B® bw 
_ 
iS) 
(oe) 
N 


Step 1. Let U) = Aandr = 1. 
Step 2. No row exchange is needed since a), # 0. 


2 6 4 4 

1 a. (00 O 1 

Step 3. Add (2)(row 1) to row 2, and (— ; )(row 1) to row 4 to get U“? = or. i : 
0 0 -1l -4 


Step 4. The submatrix in rows 2, 3, 4 is not zero, so let r = 2 and return to Step 2. 
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Step 2. Search the submatrix in rows 2 to 4 of U®) to see that its first nonzero column is column 3 
and the first nonzero entry in this column is in row 3 of U®). Exchange rows 2 and 3 in U to get 


2 6 4 4 
0 0 1 4 
(3) 
eS 0 0 0 1 
0 0 -1 -4 
2 6 4 4 
: 0 0 1 4 
Step 3. Add row 2 to row 4 in U®) to get U = ote te 
0 0 0 0 


Step 4. Now U™ is in REE, so Gaussian Elimination is finished. 
Step 5. The pivot positions are (1, 1), (2,3), and (3, 4). Add —4(row 3) to rows 1 and 2 of U™) to get 


2 6 4 0 2 6 0 0 
. 0 0 1 0 0 0 1 0 
Gy — (5) (5) (6) _ 
UY = ane 1| Add 4@ow 2) of U to row 1 of U“’ to get US? = 0-6 Oa: 
0 0 0 0 0 0 0 0 
13 0 0 
. Bredesen ey VO" OAD JOM des 
Step 6. Multiply row 1 of U‘”’ by 5, obtaining U'? = 0001 , which is RREF(A). 
0 0 0 0 
1.4 Systems of Linear Equations 
Definitions: 
A linear equation is an equation of the form a,x, +--+ -+apxp = bwhered),...,a),b € F and x,,...,Xp 


are variables. The scalars a; are coefficients and the scalar b is the constant term. 
A system of linear equations, or linear system, is a set of one or more linear equations in the same 
AX, +++ +A pXp = Dy 


: Ag|X2 + +++ + AypXp = bg 
variables, such as PvP 


. A solution of the system is a p-tuple (c),...,¢») such that 


Am X + +++ + AmpXp = bin 
letting x; = c; for each j satisfies every equation. The solution set of the system is the set of all solutions. A 
system is consistent if there exists at least one solution; otherwise it is inconsistent. Systems are equivalent 
if they have the same solution set. If b; = 0 for all j, the system is homogeneous. A formula that describes 
a general vector in the solution set is called the general solution. 


Ay1X1 +++ + apXp = Dy an Ap 
For the system ae Leb Bape =U ,themx pmatrixA=]|] : .,, +: | isthe coefficient 
Ami X, + +++ + AmpXp = Vm Gm *** Amp 
by x) 
matrix, b = : | is the constant vector, and x = : | is the unknown vector. The m x (p + 1) matrix 
bin Xp 


[A b] is the augmented matrix of the system. It is customary to identify the system of linear equations 
C1 
with the matrix-vector equation Ax = b. This is valid because a column vector x= | : | satisfies Ax = 


c 
P 
b if and only if (c;,..., cp) is a solution of the linear system. 
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Observe that the coefficients of x; are stored in column k of A. If Ax = b is equivalent to Cx = d and 
column k of C is a pivot column, then x; is a basic variable; otherwise, x; is a free variable. 


Facts: 


Let Ax = b bea linear system, where A is an m x p matrix. 


1. [SIFOO, pp. 27, 118] Ifelementary row operations are done to the augmented matrix [ A b], obtaining 
a new matrix [C d], the new system Cx = d is equivalent to Ax = b. 

2. [SIFOO, p. 24] There are three possibilities for the solution set of Ax = b: either there are no solutions 
or there is exactly one solution or there is more than one solution. If there is more than one solution 
and F is infinite (such as the real numbers or complex numbers), then there are infinitely many 
solutions. If there is more than one solution and F is finite, then there are at least | F| solutions. 

3. A homogeneous system is always consistent (the zero vector 0 is always a solution). 


nN 


. The set of solutions to the homogeneous system Ax = 0 is a subspace of the vector space F ?. 

. [SIFOO, p. 44] The system Ax = b is consistent if and only if b is not a pivot column of [A b], that 
is, if and only if rank([A b]) = rank A. 

6. [SIFOO, pp. 29-32] Suppose Ax = b is consistent. It has a unique solution if and only there is a 

pivot position in each column of A, that is, ifand only if there are no free variables in the equation 


on 


Ax = b. Suppose there are t > 1 nonpivot columns in A. Then there are t free variables in the 
system. If RREF([A b]) = [C dl], then the general solution of Cx = d, hence of Ax = b, can 
be written in the form x = s,vj + -:: + s;v; + w where vj,..., V;; Ware column vectors and 
Sj,... 5S, are parameters, each representing one of the free variables. Thus x = w is one solution of 
Ax =b. Also, the general solution of Ax = 0 isx = syv) +--+ + 5;¥;. 

7. [SIFOO, pp. 29-32] (General solution of a linear system algorithm) 


Algorithm 2: General Solution of a Linear System Ax = b 


This algorithm is intended for small systems using rational arithmetic. It is not the most efficient and 
when some pivots are relatively small, using this algorithm in floating point arithmetic can yield inaccu- 
rate results. (For more accurate and efficient algorithms, see Chapter 38.) Let A €¢ F”*? andbe F?*'. 


1. Calculate RREF([A b]), obtaining [C d]. 
2. If there is a pivot in the last column of [C d], stop. There is no solution. 
3. Assume the last column of [C d] is not a pivot column, and let d = [d),...,dn]". 
a. Ifrank(C) = p, so there exists a pivot in each column of C, then x= dis the unique solution 
of the system. 
b. Suppose rank C =r < p. 

i. Write the system of linear equations represented by the nonzero rows of [C d]. In each 
equation, the first nonzero term will be a basic variable, and each basic variable appears 
in only one of these equations. 

ii. Solve each equation for its basic variable and substitute parameter names for the p — r 
free variables, say s,,...,5p--. This is the general solution of Cx = d and, thus, the 
general solution of Ax = b. 

iii. To write the general solution in vector form, as x = s w+... +sp_,viP-) +w, let (i, k;) 
be thei pivot position of C. Definewe F? byw;, = dj fori = 1,...,7,andall other en- 
tries of ware 0. Let Xu; be the j th free variable, and define the vectors v) € F? as follows: 


For j =1,...,p—1, 
the u;-entry of vis 1, 
fori = 1,...,7r, the k;-entry of v'/) is —Ciu;> 
and all other entries of v') are 0. 
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Examples: 
‘ =0 : 110 ee AO 
1. Thelinear system me : = has augmented matrix E ll I . The RREF of this is | G4 A ) 
which is the augmented matrix for the equivalent system e - . Thus, the original system has a 
2 = 


1.5 


. These systems have no solutions: 


: sued et 8 0 
unique solution in R’, (0,0). In vector form the solution is x = 4 — | . 


+x, =2 ; ee 1 
The system a: has a unique solution in R’, (1,1),orx= eo : 
xX} —x»»=0 x2 1 
Xp +ty+2%3=2 0 
. The system X_ + x3 = 2 has a unique solution in R’, (0, 2,0), orx= |2 
x3 = 0 0 
xX, + x2= 2 F - . : 2 : 
. The system has infinitely many solutions in R°. The augmented matrix reduces 
2x, + 2x, =4 


1 1 2 ; , \ ; . : 
to | 66 4 , so the only equation left is x; + x. = 2. Thus x, is basic and x, is free. Solving 
x) =—s+2 


x2= 


for x; and letting x. = s gives x} = —s + 2. Then the general solution is , or all 


vectors of the form (—s + 2,5). Letting x = 4 , the vector form of the general solution is 
2 


“Pete 


Xp +x. +x3+%x%,= 1 


. The system has infinitely many solutions in R*. Its augmented matrix 
X2 +x3— x4 = 3 
1141 1 1 1 0 0 2 —2 ; : 
| a reduces to | a ae | . Thus, x; and x, are the basic variables, and 
x3 and x, are free. Write each of the new equations and solve it for its basic variable 
xyp= —2x4 —2 : 
to see . Let x3 = $s, and xy = 5s» to get the general solution 
Xp = —%3 +X, +3 : : : 3 6 6 
xj = —2s,—-2 0 —2 —2 
=— 3 -—l 1 3 
es sorx = s;v 4 sv) +we sy +58) + 
x3 = S| 1 0 0 
X4 = $2 0 1 0 


xX) +x%y»+%3;=0 
and x; — x. — x3 = 0. This can be verified by 
x27 x3 = 1 


x) +x»=0 
Xj +xy=1 


inspection, or by calculating the RREF of the augmented matrix of each and observing that each 
has a pivot in its last column. 


Matrix Inverses and Elementary Matrices 


Invertibility is a strong and useful property. For example, when a linear system Ax = b has an invertible 
coefficient matrix A, it has a unique solution. The various characterizations of invertibility in Fact 10 
below are also quite useful. Throughout this section, F will denote a field. 
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Definitions: 


Ann x n matrix A is invertible, or nonsingular, if there exists another n x n matrix B, called the inverse 
of A, such that AB = BA = I,,. The inverse of A is denoted A7! (cf. Fact 1). If no such B exists, A is not 
invertible, or singular. 
For an n x n matrix anda positive integer m, the mth power of Ais A” = AA... A. Itisalso convenient 
—~— 


m copies of A 


to define A° = I,,. If A is invertible, then A~” = (A7!)”. 
An elementary matrix is a square matrix obtained by doing one elementary row operation to an identity 
matrix. Thus, there are three types: 


1. A multiple of one row of I, has been added to a different row. 
2. Two different rows of I,, have been exchanged. 
3. One row of I, has been multiplied by a nonzero scalar. 


Facts: 


1. [SIFOO, pp. 114-116] If A € F"*” is invertible, then its inverse is unique. 

2. [SIFOO, p. 128] (Method to compute A~!) Suppose A € F"*". Create the matrix [A I,,] and 
calculate its RREF, which will be of the form [RREF(A)X]. If RREF(A) = [I,, then A is invertible 
and X = A7!. If RREF(A) 4 I,, then A is not invertible. As with the Gaussian algorithm, this 
method is theoretically correct, but more accurate and efficient methods for calculating inverses 
are used in professional computer software. (See Chapter 75.) 

3. [SIFOO, pp. 114-116] If A € F”*" is invertible, then A™! is invertible and (A~!)~! = A. 

4, [SIFOO, pp. 114-116] If A,B © F"*" are invertible, then AB is invertible and (AB)~! = 

BlAT, 

5. [SIFOO, pp. 114-116] If A € F”*" is invertible, then A’ is invertible and (A?)~! = (A7!)!. 

6. If A € F”*" is invertible, then for each b € F"*!, Ax= b has a unique solution, and it isx = A~'b. 

7. [SIFOO, p. 124] If A € F"*" and there exists C € F"*” such that either AC = I, or CA = I, then 

Ais invertible and A~! = C. That is, a left or right inverse for a square matrix is actually its unique 

two-sided inverse. 

8. [SIFOO, p. 117] Let E be an elementary matrix obtained by doing one elementary row operation to 

I,. If that same row operation is done to an n x p matrix A, the result equals EA. 

9. [SIFOO, p. 117] An elementary matrix is invertible and its inverse is another elementary matrix of 

the same type. 

10. [SIFOO, pp. 126] (Invertible Matrix Theorem) (See Section 2.5.) When A € F””", the following 

are equivalent: 


¢ Ais invertible. 

¢ RREF(A) = In. 

¢ Rank(A) =n. 

* The only solution of Ax = 0 is x= 0. 

* For every b € F™*!, Ax =b has a unique solution. 

* For every b €¢ F"*!, Ax =b has a solution. 

¢ There exists B € F"*" such that AB = I,,. 

¢ There exists C € F"*" such that CA = I,. 

¢ A’ is invertible. 

* There exist elementary matrices whose product equals A. 


11. [SIFOO, p. 148] and [Lay03, p.132] Let A € F"*” be upper (lower) triangular. Then A is invertible 
if and only if each diagonal entry is nonzero. If A is invertible, then A7' is also upper (lower) 
triangular, and the diagonal entries of A~! are the reciprocals of those of A. In particular, if L isa 
unit upper (lower) triangular matrix, then L~! is also a unit upper (lower) triangular matrix. 
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12. 


Matrix powers obey the usual rules of exponents, i.e, when A* and A’ are defined for integers 
s and t, then ASA’ = AS*’, (AS)! = A™, 


Examples: 


1. 


1.6 


. The matrix A = E 


For any n, the identity matrix I, is invertible and is its own inverse. If P is a permutation matrix, 
it is invertible and P-! = P?. 


2 1 
Ar} = RB. 
0.2 4 1 5 —-10 —-5 


2 7 


A= ; i and B = E | , then calculation shows AB = BA = I, so A is invertible and 


.TIfA= 0 2 1|,then A7'= ]0 0.5 0.5] , as can be verified by multiplication. 


0 0 -!l 0 oO -!l 


, A is not invertible since RREF(A) # Jy. Alternatively, if B is any 2 x 2 


matrix, AB is of the form | - | , which cannot equal Ib. 


. Let A be an n Xx n matrix A with a zero row (zero column). Then A is not invertible since 


RREF(A) # I,. Alternatively, if B is any n x n matrix, AB has a zero row (BA has a zero column), 
so B is not an inverse for A. 


.IfA= k 4 is any 2 x 2 matrix, then A is invertible if and only if ad — bc ¥ 0; further, when 


ad — bc \—c 


(The determinant is defined for any n x n matrix in Section 4.1.) Using this formula, the matrix 


1 = 
ad—be#0,A' = | G . The scalar ad — bc is called the determinant of A. 


7 3 ; ae : 1 -—3 
A= ; | from Example 2 (above) has determinant I, so A is invertibleand A~! = _ , ‘| . 


[tf 2 ; . 3 : ; : 
as noted above. The matrix | , | from Example 3 (above) is not invertible since its determinant 


4 
is 0. 
1 3 0 1 0 7 —3 O 
Let A= |2 7 O|.Then RREF({[A [,]) = |0 1 1 0], so Aq! exists and 
111 0 O —5 2 1 


equals | —2 1 0 
—5 2 ot 


LU Factorization 


This section discusses the LU and PLU factorizations of a matrix that arise naturally when Gaussian 
Elimination is done. Several other factorizations are widely used for real and complex matrices, such as 
the QR, Singular Value, and Cholesky Factorizations. (See Chapter 5 and Chapter 38.) Throughout this 
section, F will denote a field and A will denote a matrix over F. The material in this section and additional 
background can be found in [GV96, Sec. 3.2]. 


Definitions: 


Let A be a matrix of any shape. 
An LU factorization, or triangular factorization, of A is a factorization A = LU where L is a square 
unit lower triangular matrix and U is upper triangular. A PLU factorization of A is a factorization of 
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the form PA = LU where P is a permutation matrix, L is square unit lower triangular, and U is upper 
triangular. An LDU factorization of A isa factorization A = LDU where L is a square unit lower triangular 
matrix, D is a square diagonal matrix, and U is a unit upper triangular matrix. 

A PLDU factorization of A is a factorization PA = LDU where P is a permutation matrix, L is a square 
unit lower triangular matrix, D is a square diagonal matrix, and U is a unit upper triangular matrix. 


Facts: [GV 96, Sec. 3.2] 


1. Let A be square. If each leading principal submatrix of A, except possibly A itself, is invertible, 
then A has an LU factorization. When A is invertible, A has an LU factorization if and only if each 
leading principal submatrix of A is invertible; in this case, the LU factorization is unique and there 
is also a unique LDU factorization of A. 

2. Any matrix A hasa PLU factorization. Algorithm 1 (Section 1.3) performs the addition of multiples 
of pivot rows to lower rows and perhaps row exchanges to obtain an REF matrix U. If instead, the 
same series of row exchanges are done to A before any pivoting, this creates PA where P is a 
permutation matrix, and then PA can be reduced to U without row exchanges. That is, there exist 
unit lower triangular matrices E; such that Ex, ... E;(PA) = U. It follows that PA = LU, where 
L = (Ex... £,)7! is unit lower triangular and U is upper triangular. 

3. In most professional software packages, the standard method for solving a square linear system 
Ax=b, for which A is invertible, is to reduce A to an REF matrix U as in Fact 2 above, choosing row 
exchanges by a strategy to reduce pivot size. By keeping track of the exchanges and pivot operations 
done, this produces a PLU factorization of A. Then A = P?’LUand P?LUx = bis the equation to 
be solved. Using forward substitution, P™ Ly = b can be solved quickly for y, and then Ux = y can 
either be solved quickly for x by back substitutution, or be seen to be inconsistent. This method 
gives accurate results for most problems. There are other types of solution methods that can work 
more accurately or efficiently for special types of matrices. (See Chapter 7.) 


Examples: 
1 1 2.3 
these 1 -1 -3 1 ; he nat 
1. Calculate a PLU factorization for A = 0 1 iat: If Gaussian Elimination is performed 
—l 0 -1 1 
on A, after adding row 1 to rows 2 and 4, rows 2 and 3 must be exchanged and the final result is 
1 1 2 3 
0 1 il . ' : 
U = E3PE,E,;A= OF igo wk where E), E>, and E; are lower triangular unit matrices and 
0 O 0 3 


P is a permutation matrix. This will not yield an LU factorization of A. But if the row exchange 


10 0 | | i. ae’ 2 3] 
; ae 0 0 1 +0 0 1 1 1 
is done to A first, by multiplying A by P = 0 10 017°" gets PA= | | Taq ae ak 
0 0 0 1 —l 0 -1 1 
then Gaussian Elimination can proceed without any row exchanges. Add row 1 to rows 3 and 4 to get 
11 2.3 1 0 0 0 1 0 0 0 
1 1 1 0 0 1 0 0 
F)F,\PA = a ee where F, = ie ae ae and F, = Oo 1° . Then add 
0 1 1 4 0 0 0 1 1 00 1 
1 1 2 3 1 0 0 0 
0 1 15c 0 1 0 0 
(—1)(row 2) to row 4 to get U = F3 F, Fj PA= oe eee ae , where F3 = 0 6-40 
0 0 0 3 0 -1 0 1 
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3. 


5. 


Note that U is the same upper triangular matrix as before. Finally, L = (F3 F F,)~! is unit lower 
triangular and PA = LU is true, so this is a PLU factorization of A. To get a PLDU factorization, 


1 0 0 0 1 1 2 3 
0 1 0 0 1 1 1 
use the same P and L, and define D = en and U = Got08 oe Sa 
0 0 3 0 0 0 ih 
1 3 4 
.LetA=LU= 1 1 5| . Each leading principal submatrix of A is invertible so A has 
2° 12 3 
both LU and LDU factorizations: 
10 0;}1 3 4 10 0;}1 0 =O 
A=LU= |-1 1 0}]0 2 —1). ThisyieldsanLDUfactorizationofA,}|—1 1 0]]0 2 0 
23 1;}0 0 —2 23 1)}0 0 —2 
1 3 4 1 
0 1 —0.5]. With the LU factorization, an equation suchas Ax= | 1] canbe solved efficiently 
0 0 1 0 
1 1 
as follows. Use forward substitution to solve Ly = | 1], getting y= | 2], and then backward 
0 —8 
—24 
substitution to solve Ux = y, getting x = 3 
4 


0 -!l 
F : : : 0 1 
Any invertible matrix whose (1, 1) entry is zero, such as | 1 | or } 1 1 1], does not have 


an LU factorization. 


1 3 4 
The matrix A= |—1 —3 -—5} is not invertible, nor is its leading principal 2 x 2 submatrix, 
2 6 6 
1 0 0 1 3 4 
but it does have an LU factorization: A= LU = |—1 1 O| {0 O —1). To find out if an 
2° 3.0 i] 0 0 1 
1 
equation such as Ax = : is consistent, notice Ly = | 1] yields y = 2), but Ux =y is 
0 —8 
inconsistent, hence Ax = has no solution. 


0 1 1 1 
O},andU=j]0 —-1 5 
1 


= 5 
The matrix A = : 1] has no LU factorization, but does have a PLU factorization with 
2 
0 
1 
1 0 QO -4 


- Oo _ 
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2.1 Span and Linear Independence 


Let V be a vector space over a field F. 


Definitions: 


A linear combination of the vectors vj, v2,..., vk € V isasum of scalar multiples of these vectors; that is, 
C1Vy +02V2 +-++++cxvx, for some scalar coefficients c),C2,...,c, € F.IfS isa set of vectors in V, a linear 
combination of vectors in S is a vector of the form c,v, + cyv2 +--+: + cgvy with k € N,v; € S,c; € F. 
Note that S may be finite or infinite, but a linear combination is, by definition, a finite sum. The zero 
vector is defined to be a linear combination of the empty set. 

When all the scalar coefficients in a linear combination are 0, it is a trivial linear combination. A sum 
over the empty set is also a trivial linear combination. 

The span of the vectors Vv), V2,..., Vx € V is the set of all linear combinations of these vectors, denoted 
by Span(v),V2,...,Vvx). If S is a (finite or infinite) set of vectors in V, then the span of S, denoted by 
Span(S), is the set of all linear combinations of vectors in S. 

If V = Span(S), then S spans the vector space V. 

A (finite or infinite) set of vectors S in V is linearly independent if the only linear combination of 
distinct vectors in S that produces the zero vector is a trivial linear combination. That is, if v; are distinct 
vectors in S and c)vyj + C2V2 +--+: + cyvy = O, then cy C2 see Ck 0. Vectors that are not 
linearly independent are linearly dependent. That is, there exist distinct vectors v,,V2,...,v, € S and 
C15C€2,..., Cx notall 0 such that cjv, + c2v2 +--+ c.v, = 0. 
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Facts: The following facts can be found in [Lay03, Sections 4.1 and 4.3]. 


1. Span(@) = {0}. 

2. A linear combination of a single vector v is simply a scalar multiple of v. 

3. Ina vector space V, Span(vj, V2,..., Vx) is a subspace of V. 

4. Suppose the set of vectors S = {v1,v2,..., Vx} spans the vector space V. If one of the vectors, say 
v;, is a linear combination of the remaining vectors, then the set formed from S by removing v; 
still spans V. 

5. Any single nonzero vector is linearly independent. 

6. Two nonzero vectors are linearly independent if and only if neither is a scalar multiple of the other. 

7. If S spans V and S C T, then T spans V. 

8. If T isa linearly independent subset of V and S C T, then S is linearly independent. 

9. Vectors V1, V2,...,Vx are linearly dependent if and only if vj = cyvy + +++ + Cj-1Vj-1 + Ci Vi41 
+-+++ cxvg, for some 1 <i < k and some scalars c),...,¢€j-1,Ci41,...,Cx. A set S of vectors in 
V is linearly dependent if and only if there exists v € S such that v is a linear combination of other 
vectors in S. 

10. Any set of vectors that includes the zero vector is linearly dependent. 
Examples: 
1. Linear combinations | ‘ Y € R’ are vectors of the form c, ! +) ‘ = ae ; 
—l 3 -—1 3 —c, + 3c, 
1 


for any scalars cj,cp € R. Any vector of this form is in span ( 


1 0 
Span ( | : , H = R’ and these vectors are linearly independent. 


2. Ifv € R” and v ¥ 0, then geometrically Span(v) is a line in R” through the origin. 


. Suppose n > 2 and yj, v2 € R” are linearly independent vectors. Then geometrically Span(vj, v2) is 


a plane in R” through the origin. 


. Any polynomial p(x) € R[x] of degree less than or equal to 2 can easily be seen to be a linear 


combination of 1, x, and x. However, p(x) is also a linear combination of 1, 1 + x, and 1+ x?. So 
Span(1, x, x?) = Span(1, 1 + x, 1 +x?) = R[x;2]. 


1 0 
0 1 0 
. The nvectorse, = |9|,e=/9],...,e, = : | span F", for any field F. These vectors are 
: 0 
0 0 1 


also linearly independent. 


1 0 1 0 1 
. In R’, i and ;) are linearly independent. However, | ‘ ;) , and |] are linearly 
1 1 
dependent, because = +2 0 : 
5 —1 3 


. The infinite set {1,x,x?,...,x",...} is linearly independent in F [x], for any field F. 
. Inthe vector space of continuous real-valued functions on the real line, C(R), the set {sin(x), sin(2x), 


...,Sin(nx), cos(x), cos(2x),...,cos(nx)} is linearly independent for any n € N. The infinite 
set {sin(x), sin(2x),...,sin(nx),...,cos(x),cos(2x),...,cos(nx),...} is also linearly independent 
inC(R). 
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Applications: 


1. The homogeneous differential equation Las 


pare 


d? dy 

dx? dx 
y2(x) = e*. Any linear combination y(x) = c,y\(x) + c2y2(x) is a solution of the differential 
equation, and so Span(e?*, e*) is contained in the set of solutions of the differential equation 
(called the solution space for the differential equation). In fact, the solution space is spanned by 
e* and e*, and so is a subspace of the vector space of functions. In general, the solution space 
for a homogeneous differential equation is a vector space, meaning that any linear combination of 
solutions is again a solution. 


2x 


+ 2y = 0 has as solutions y|(x) = e** and 


Basis and Dimension of a Vector Space 


Let V be a vector space over a field F. 


Definitions: 


A set of vectors 6 in a vector space V is a basis for V if 


* Bisa linearly independent set, and 


* Span(B) = V. 
1 0 0 
0 1 0 
The set E€, = 4e, = |9},e=]9],...,e,= : is the standard basis for F”. 
0 
0 0 1 


The number of vectors in a basis for a vector space V is the dimension of V, denoted by dim(V). If a 
basis for V contains a finite number of vectors, then V is finite dimensional. Otherwise, V is infinite 
dimensional, and we write dim(V) = oo. 


Facts: All the following facts, except those with a specific reference, can be found in [Lay03, Sections 4.3 
and 4.5]. 


1. 
2: 
3. 


Every vector space has a basis. 

The standard basis for F” is a basis for F”, and so dim F” = n. 

A basis B in a vector space V is the largest set of linearly independent vectors in V that contains B, 
and it is the smallest set of vectors in V that contains B and spans V. 

The empty set is a basis for the trivial vector space {0}, and dim({0}) = 0. 

If the set S = {v,,...,Vp} spans a vector space V, then some subset of S forms a basis for V. In 
particular, if one of the vectors, say v;, is a linear combination of the remaining vectors, then the 
set formed from S by removing v; will be “closer” to a basis for V. This process can be continued 
until the remaining vectors form a basis for V. 

If S is a linearly independent set in a vector space V, then S can be expanded, if necessary, to a basis 
for V. 

No nontrivial vector space over a field with more than two elements has a unique basis. 

If a vector space V has a basis containing n vectors, then every basis of V must contain n vectors. 
Similarly, if V has an infinite basis, then every basis of V must be infinite. So the dimension of V 
is unique. 

Let dim(V) = n and let S be a set containing n vectors. The following are equivalent: 

* Sisa basis for V. 

* S spans V. 


* S is linearly independent. 
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10. Ifdim(V) = n, then any subset of V containing more than n vectors is linearly dependent. 

11. Ifdim(V) = n, then any subset of V containing fewer than n vectors does not span V. 

12. [Lay03, Section 4.4] If B = {b;,...,b,} is a basis for a vector space V, then each x € V can be 
expressed as a unique linear combination of the vectors in 8. That is, for each x € V there is a 


unique set of scalars c),¢2,...,¢p such that x = cb; + cobz +--+ + Cpbp. 
Examples: 
5 1 0 : . 2 . 2 
1. InR’, i and 5 [ane linearly independent, and they span R*. So they form a basis for R* and 
dim(R?) = 2. 
2. In F [x], the set {1, x, x*,...,x"} isabasis for F [x;n] foranyn € N. The infinite set {1, x, x”, x°,...} 


is a basis for F [x], meaning dim(F [x]) = oo. 
3. The set of m x n matrices Ej having a 1 in the i, j-entry and zeros everywhere else forms a basis 
for F'*", Since there are mn such matrices, dim(F™*") = mn. 


1 0 1 
4. Theset S = H : | ; | ; ;] clearly spans IR’, but it is not a linearly independent set. However, 


removing any single vector from S will cause the remaining vectors to be a basis for R?, because 
any pair of vectors is linearly independent and still spans R’. 


1 0 
1 0 ee ‘ ; : ae : 
5. The set S = aa et is linearly independent, but it cannot be a basis for R* since it does 
0 1 
not span R*. However, we can start expanding it to a basis for R* by first adding a vector that is not 
1 
: 0 : : 4 
in the span of S, such as él Then since these three vectors still do not span R*, we can adda 
0 


vector that is not in their span, such as . These four vectors now span R* and they are linearly 


oro°o 


independent, so they form a basis for R‘. 
6. Additional techniques for determining whether a given finite set of vectors is linearly independent 
or spans a given subspace can be found in Sections 2.5 and 2.6. 


Applications: 
1. Because y;(x) = e?* and y(x) = e* are linearly independent and span the solution space for the 
7 d 
homogeneous differential equation va —3 ~ + 2y = 0, they form a basis for the solution space 
x x 


and the solution space has dimension 2. 


2.3 Direct Sum Decompositions 


Throughout this section, V will be a vector space over a field F , and Wj, fori = 1,...,k, will be subspaces 
of V. For facts and general reading for this section, see [HK71]. 
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Definitions: 

The sum of subspaces W,, for i = 1; .cikvis SW =Wet+---+wme={wit---twel|w € Wi}. 
The sum W, + --- + W, is a direct sum if for alli = 1,...,k, we have WM Ys WwW; = {0}. 
W = W, @--- © W% denotes that W = W, +--- + W, and the sum is direct. The subspaces W,, 


fori = i,...,k, are independent if for w; € W;, wi +----++ wk = Oimplies w; = 0 for alli = 1,...,k. Let 
V;, fori = 1,...,k, be vector spaces over F. The external direct sum of the V;, denoted V, x --- x Vj, is 
the cartesian product of V;, fori = 1,...,k, with coordinate-wise operations. Let W be a subspace of V. 


An additive coset of W isa subset of the form v + W = {v+-w|w € W}withv € V. The quotient of V by 
W, denoted V/W, is the set of additive cosets of W with operations (v; + W) + (v2+ W) = (v; + v2) +W 
and c(v + W) = (cv) + W, for any c € F. Let V = W@ U, let By and By be bases for W and U 
respectively, and let B = By U By. The induced basis of B in V/ W is the set of vectors {u+ W|u € By}. 


Facts: 


1. W= W, © Wy if and only if W = W, + W, and W,N W, = {0}. 
2. If Wis a subspace of V, then there exists a subspace U of V such that V = W@ U. Note that U is 
not usually unique. 
3. LetW=W,+.---+ W. The following are equivalent: 
*“W=W,4:::O WM. That is, for alli = 1,...,k, we have W; Ge ares W; = {0}. 
+ WN Dia) W; = {0}, forall i = 2,...,k. 
* For each w € W, wean be expressed in exactly one way as a sum of vectors in Wi,..., Wy. That 
is, there exist unique w; € W,, such that w = w; +--- + wk. 


¢ The subspaces W;, fori = 1,...,k, are independent. 
¢ If 6; is an (ordered) basis for W,, then B = eS B; is an (ordered) basis for W. 


4. If B is a basis for V and B is partitioned into disjoint subsets B;, for i = 1,...,k, then 
V = Span(B,) ® --- @ Span(B,). 
5. If Sisalinearly independent subset of V and S is partitioned into disjoint subsets S;, fori = 1,...,k, 


then the subspaces Span(S;),..., Span(S;) are independent. 
6. If V is finite dimensional and V = W, + --- + Wj, then dim(V) = dim(W,) +--+ + dim(W,) if 
and only ifV = W, ®---® WM. 
7. Let Vj, fori = 1,...,k, be vector spaces over F. 
° Vi x--- x V, isa vector space over F. 
° V; = {(0,...,0,v;,0,...,0) | vi € Vi} (where v; is the ith coordinate) is a subspace of 
Vix-+ xk Vy. 
° Vix X WH V0 OK. 
¢ If V;, fori = 1,...,k, are finite dimensional, then dim V; = dim V; and dim(V, x --- x Vi) = 
dim V, +---+ dim Vy. 


8. If Wis a subspace of V, then the quotient V/W is a vector space over F. 
9. Let V = W@ U, let By and By be bases for W and U respectively, and let 6 = By U By. The 
induced basis of 6 in V/W is a basis for V/W and dim(V/W) = dim U. 


Examples: 
1. Let B = {vj,...,v,} bea basis for V. Then V = Span(v,) @ --- @ Span(v,,). 


2. Let X = jer} y = {°] iver}, and 2 = 14 jeer}, then = 
y Zz 


X®Y=YO@OZ=X@9@Z. 


x 
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3. In F"*", let W, be the subspace of symmetric matrices and W, be the subspace of skew-symmetric 
A+A™ A-—A?™ A+ AT 

matrices. Clearly, W, MN W2 = {0}. Forany A € F"*", A= ; + a, where - € 

T 


W, and € W). Therefore, F"*" = W, ® Wy. 


4. Recall that the function f € C(R) isevenif f(—x) = f(x) forallx,and f isoddif f(—x) = — f(x) 
for all x. Let W; be the subspace of even functions and W) be the subspace of odd functions. 
Clearly, Wi A W, = {0}. For any f € C(R), f = fi + fo, where fi(x) = faye son) 

F(x) = f(=*) 

2 

5. Given a subspace W of V, we can find a subspace U such that V = W@ U by choosing a basis 

for W, extending this linearly independent set to a basis for V, and setting U equal to the span of 


ew 


and f(x) = € W). Therefore, C(R) = W, ® W. 


a 1 
the basis vectors not in W. For example, in R?, Let W = —2a| |aeR>.Ifw= | -2], 
a 1 


then {w} is a basis for W. Extend this to a basis for R*, for example by adjoining e; and e). Thus, 
V = W@U, where U = Span(ej,e2). Note: there are many other ways to extend the basis, and 


many other possible U. 
+3) x?+4x-2 aia |B 
"1-1 0) | 


| ey = 


| 1 \ respectively. Then Byy = Bx U By and Byz = Bx U Bz are bases for R’. In R*/ X, the 


2 


6. Inthe external direct sum R[x;2] x R?*?, (2» +7, i 


) 


7. The subspaces X, Y, Z of R? in Example 2 have bases By = H By = ; 


2) 


(se + 12x +1, i 


0 
induced bases of Byy and Byz are : +X and i +X \ respectively. These are equal 
0 1 0 
b X= X= X. 
ecause 1 + 1 =F 0 + 1 =e 


2.4 Matrix Range, Null Space, Rank, 
and the Dimension Theorem 


Definitions: 


For any matrix A € F”*", the range of A, denoted by range(A), is the set of all linear combinations of 
the columns of A. If A = [m, m, ... m,], then range(A) = Span(m,,mp,...,m,,). The range of A is 
also called the column space of A. 

The row space of A, denoted by RS(A), is the set of all linear combinations of the rows of A. If 
A= [v, V2 ... Vm)", then RS(A) = Span(v),V2,..., Vm): 

The kernel of A, denoted by ker(A), is the set of all solutions to the homogeneous equation Ax = 0. 
The kernel of A is also called the null space of A, and its dimension is called the nullity of A, denoted by 
null(A). 

The rank of A, denoted by rank(A), is the number of leading entries in the reduced row echelon form 
of A (or any row echelon form of A). (See Section 1.3 for more information.) 
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A, B € F™*" are equivalent if B = Cy; AC, for some invertible matrices C) € F™*™” and Cy € F"™". 
A,B € F"*" are similar if B = C~!AC for some invertible matrix C € F"*". For square matrices 


Ay € F™*",..., Ay € F™*"*, the matrix direct sum A = A; ®--- @ Ag, is the block diagonal matrix 
Aj 0 r 
with the matrices A; down the diagonal. That is, A= om , where A € F"*" with n = Sti 
i=l 
0 Ax 


Facts: Unless specified otherwise, the following facts can be found in [Lay03, Sections 2.8, 4.2, 4.5, 
and 4.6]. 


1. The range of an m x n matrix A is a subspace of F”. 

2. The columns of A corresponding to the pivot columns in the reduced row echelon form of A 
(or any row echelon form of A) give a basis for range(A). Let v1,v2,...,v, € F”™. If matrix 
A= [v, V2 ... vx], then a basis for range(A) will be a linearly independent subset of vj, v2,... , VE 
having the same span. 

3. dim(range(A)) = rank(A). 

4, The kernel of an m x n matrix A is a subspace of F”. 

5. If the reduced row echelon form of A (or any row echelon form of A) has k pivot columns, then 
null(A) =n—k. 

6. If two matrices A and B are row equivalent, then RS(A) = RS(B). 

7. The row space of an m x n matrix A is a subspace of F”. 

8. The pivot rows in the reduced row echelon form of A (or any row echelon form of A) give a basis 
for RS(A). 

9. dim(RS(A)) = rank(A). 

10. rank(A) = rank(A‘). 

11. (Dimension Theorem) For any A € F”*", n = rank(A) + null(A). Similarly, m = dim(RS(A)) + 
null(A7). 

12. Avectorb € F” isin range( A) ifand only ifthe equation Ax = bhasasolution. So range(A) = F” 
if and only if the equation Ax = b has a solution for every b € F™. 

13. A vector a € F” is in RS(A) if and only if the equation Aly = ahasa solution. So RS(A) = F" if 
and only if the equation A’y = a has a solution for every a € F". 

14. Ifais a solution to the equation Ax = b, then a + vis also a solution for any v € ker(A). 

15. [HJ85, p. 14] If A € F’”*" is rank 1, then there are vectors v € F”™ andu € F” so that A = vu. 

16. If A € F”™*" is rank k, then A is a sum of k rank 1 matrices. That is, there exist A,,..., A, with 
A= A; +---+ Ax and rank(A;) = 1, fori = 1,...,k. 

17. [HJ85, p. 13] The following are all equivalent statements about a matrix A € F”™”, 
(a) The rank of A is k. 

(b) dim(range(A)) = k. 

(c) The reduced row echelon form of A has k pivot columns. 

(d) A row echelon form of A has k pivot columns. 

(e) The largest number of linearly independent columns of A is k. 
(f) The largest number of linearly independent rows of A is k. 

18. [HJ85, p. 13] (Rank Inequalities) (Unless specified otherwise, assume that A, B € F*".) 
(a) rank(A) < min(m, n). 
(b) Ifa new matrix B is created by deleting rows and/or columns of matrix A, then rank(B) < 

rank(A). 

(c) rank(A + B) < rank(A) + rank(B). 
(d) If Ahasa p x q submatrix of 0s, then rank(A) < (m— p)+ (n—4q). 
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(e) If A €¢ F™™** and B € F**", then 
rank(A) + rank(B) — k < rank(AB) < min{rank( A), rank(B)}. 
19. [HJ85, pp. 13-14] (Rank Equalities) 

(a) If Ae C”*", then rank(A*) = rank(A’) = rank(A) = rank(A). 

(b) If A € C”*", then rank(A* A) = rank(A). If A € R”*", then rank(A?’ A) = rank(A). 

(c) Rank is unchanged by left or right multiplication by a nonsingular matrix. That is, if A € F"*" 
and B € F”*" are nonsingular, and M € F™*", then 

rank(AM) = rank(M) = rank(MB) = rank(AMB). 

(d) If A,B € F™*", then rank(A) = rank(B) if and only if there exist nonsingular matrices 
X € F™*" and Y € F"*" such that A= X BY (ie., ifand only if A is equivalent to B). 

(e) If A € F”*" has rank k, then A = XBY, for some X ¢ F™**, Y © F**", and nonsingular 
Bere, 

(f) If Ay € F™*",..., Ap © F™*"*, then rank(A; ®--- @ Ay) = rank(A,) + --- + rank(A,). 

20. Let A, B € F"*" with A similar to B. 

(a) Ais equivalent to B. 
(b) rank(A) = rank(B). 
(c) tA=tr B. 

21. Equivalence of matrices is an equivalence relation on F™™*”. 

22. Similarity of matrices is an equivalence relation on F"*". 

23. If Ae F”*" and rank(A) = k, then A is equivalent to ie ; , and so any two matrices of the 
same size and rank are equivalent. 

24. (For information on the determination of whether two matrices are similar, see Chapter 6.) 

25. [Lay03, Sec. 6.1] If A € R"*", then for any x € RS(A) and any y ¢€ ker(A), x’y = 0. So the 
row space and kernel of a real matrix are orthogonal to one another. (See Chapter 5 for more on 
orthogonality.) 

Examples: 
1 7-2 a+ 7b —2c 1 a 
1. IfA=]0-1 1] € R**?,then any vector of the form —b+ec 0- 1 || b 
2 13-3 2a + 13b — 3c 2 13-3 || ¢ 
oe: 229 
is in range(A), for any a,b,c € R. Since a row echelon form of Ais | 0 1 —11], we know that 
0 0 0 
1 7 1 0 
the set O},]-1 is a basis for range(A), and the set 71, 1 is a basis for 
2 13 —2 —l 
1 0 5 0 
RS(A). Since its reduced row echelon formis |0 1 —1 |, the set O|, 1 is another 
0 0 0 5 1 


basis for RS(A). 
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1 Piva) 
2.IfA= |0 —-1 1| € R®*?, then using the reduced row echelon form given in the previ- 
2 13 —3 
=) 
ous example, solutions to Ax = 0 have the form x = c 1], for any c € R. So ker(A) = 
1 
=5 
Span 1 
1 


1 0 3-0 2 
3. If A € R®*> has the reduced row echelon form |0 1 -—2 0 7 |, then any solution to 


Ax = 0 has the form 


—3 =2. 
2 =/ 
xX=C) 1 + C2 0 
0 1 
0 1 
for some C1, C2 € R. So, 
—3 2 
2 —7 
ker(A) = Span 1],] 0 
0 1 
0 1 
1 7 
4, Example 1 above shows that O},)-1 is a linearly independent set having the same span 
2 13 
1 7 =2 
as the set O|,/—-l1], 1 : 
2 13 —3 
1 DN teed 37 —46 
5. ; J is similar to Hi =: b 


2.5 Nonsingularity Characterizations 


From the previous discussion, we can add to the list of nonsingularity characterizations of a square matrix 
that was started in the previous chapter. 


Facts: The following facts can be found in [HJ85, p. 14] or [Lay03, Sections 2.3 and 4.6]. 


1. If Ae F”*", then the following are equivalent. 
(a) A is nonsingular. 
(b) The columns of A are linearly independent. 
(c) The dimension of range(A) is n. 
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(d) The range of A is F”. 
(e) The equation Ax = b is consistent for each b € F”. 
(f) Ifthe equation Ax = b is consistent, then the solution is unique. 
(g) The equation Ax = b has a unique solution for each b € F”. 
(h) The rows of A are linearly independent. 
(i) The dimension of RS(A) is n. 
(j) The row space of A is F”. 
(k) The dimension of ker(A) is 0. 
(1) The only solution to Ax = 0 is x = 0. 
(m) The rank of A is n. 


(n) The determinant of A is nonzero. (See Section 4.1 for the definition of the determinant.) 


2.6 Coordinates and Change of Basis 


Coordinates are used to transform a problem in a more abstract vector space (e.g., the vector space of 
polynomials of degree less than or equal to 3) to a problem in F”. 
Definitions: 


Suppose that 6 = (b;,b2,...,b,,) is an ordered basis for a vector space V over a field F and x € V. The 
coordinates of x relative to the ordered basis GB (or the B-coordinates of x) are the scalar coefficients 


C1,€2)...5€, € F such that x = c)x, + CyX. +--+ + C,X,. Whenever coordinates are involved, the 
vector space is assumed to be nonzero and finite dimensional. 
If c),C2,...5C, are the B-coordinates of x, then the vector in F”, 
Cl 
C2 
[x]z = ; 
Cn 


is the coordinate vector of x relative to 6 or the B-coordinate vector of x. 

The mapping x — [x], is the coordinate mapping determined by 5. 

If B and B’ are ordered bases for the vector space F”, then the change-of-basis matrix from B to B’ is 
the matrix whose columns are the B’-coordinate vectors of the vectors in B and is denoted by [I]. Such 
a matrix is also called a transition matrix. 


Facts: The following facts can be found in [Lay03, Sections 4.4 and 4.7] or [HJ85, Section 0.10]: 


1. For any vector x € F” with the standard ordered basis €,, = (e1,€2,...,€,), we have x = [x]¢,. 

2. For any ordered basis B = (b;,...,b,) of a vector space V, we have [b;]z = ej. 

3. If dim(V) = n, then the coordinate mapping is a one-to-one linear transformation from V onto 
F". (See Chapter 3 for the definition of linear transformation.) 

4. If Bis an ordered basis for a vector space V and vj), v2 € V,thenv; = v2 ifand onlyif [vj] = [vo]z. 

5. Let V be a vector space over a field F, and suppose B is an ordered basis for V. Then for any 
XVp...,Ve € Vandcy,...,ce € F,x = civ) +--+ + cxvy if and only if [x]g = cy[vwiJge+---+ 
CklVkl]g. So, for any x,v),...,V, € V, x € Span(v,,...,v,) if and only if [x]g € Span([vi]g,..., 
[vk ]). 

6. Suppose B is an ordered basis for an n-dimensional vector space V over a field F andwy,..., vz € V. 
The set S = {v),..., vi} is linearly independent in V if and only if the set S’ = {[vi]z,..., [vkla} 
is linearly independent in F”. 
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7. Let V be a vector space over a field F with dim(V) = n, and suppose B is an ordered basis for V. 
Then Span(v),v2,...,V%) = V for some vj, v2,...,v% € V if and only if Span([v)]g, [valg,..., 
[vilp) = FF". 

8. Suppose 6 is an ordered basis for a vector space V over a field F with dim(V) = n, and let 
S = {v1,...,Vy} be a subset of V. Then S is a basis for V if and only if {[vy]s,...; [Vn]g} is a basis 
for F” if and only if the matrix [[v)]g,..., [v,]s] is invertible. 

9. If B and 6’ are ordered bases for a vector space V, then [x]w = »[I]z [x]g for any x € V. 
Furthermore, g’ [I] is the only matrix such that for any x € V, [x]g = B [I] e [x]z. 

10. Any change-of-basis matrix is invertible. 

11. If B is invertible, then B is a change-of-basis matrix. Specifically, if B = [b, --- b,] € F”*", then 
B = ¢,[I]g, where B = (bj,...,b,,) is an ordered basis for F”. 

12. If B = (b;,...,b,) is an ordered basis for F", then ¢ [I]g = [b; --- by]. 

13. If B and B’ are ordered bases for a vector space V, then g[I] = (g[I]g)7!. 

14. If B and GB’ are ordered bases for F”, then g [I]g = (pe [I]e, )(¢, 1] ). 


Examples: 


Ll. If p(x) = ayx™ + ay_yx""| +--+» + ayx + ao € F[x3n] with the standard ordered basis 


B= (1,x,x?,...,x«"), then [p(x)]z = 
an 


2. The set B = ( | i : He forms an ordered basis for R?. If €) is the standard ordered basis 
1 


for IR, then the change-of-basis matrix from B to €) is ¢,[T]g = | ; 


wie Uo 


0 
| and (¢,[T]g)7! = 
1 0 3]. : -1 
1 .So forv = i in the standard ordered basis, we find that [v]z = (¢,[T]g)" v= F 
0 
als 


1 
3 3 

1 

3. The set B’ = (1,1+.x, 1+’) is an ordered basis for R[x; 2], and using the standard ordered basis 


3 
To check this, we can easily see that v = | 1 =3 | : + : 


111 a ee 
B = (1,x,x?) for R[x;2] we have g[P]xe = |0 1 0|.So0,(s[P]e)"' = |0 1° 0 
0 0 1 0 0 1 
5 4 
and [5 — 2x + 3x7]p = (g[P]g)7! | -2| = | —2 |. Of course, we can see 5 — 2x + 3x? = 
3 3 


4(1) — 20. + x) +3(1 + x?). 


4. If we want to change from the ordered basis B; = ( | ; | ; | : 


in R? to the ordered basis B, 


2 
(| | ; A iF then the resulting change-of-basis matrix is g,[T]s, = (¢,[T]e,)~'(e,[T]s,) = 


Ped [a 3)-[2 =} 
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5. Let S = {5 — 2x + 3x*,3 — x + 2x’, 


+ 3x} in R[x;2] with the standard ordered basis 6 = 


8 
8 
(1, x, x). The matrix A = | —2 3 | contains the B-coordinate vectors for the polynomials 
0 
3 
1 


31]. Since this row echelon form shows that A is 
1 


nonsingular, we know by Fact 8 above that S is a basis for R[x; 2]. 


3 
—l 
2 
5 
in S and it has row echelon form 


2.7 Idempotence and Nilpotence 


Definitions: 


Ais an idempotent if A? = A. 
Ais nilpotent if, for some k > 0, Ak =0. 


Facts: Allofthe following facts except those with a specific reference are immediate from the definitions. 
1. Every idempotent except the identity matrix is singular. 
2. Let A € F"*". The following statements are equivalent. 
(a) Ais an idempotent. 
(b) I — Ais an idempotent. 
(c) Ifv € range(A), then Av = v. 
(d) F” = ker A@ rangeA. 


(e) [HJ85, p. 37 and p. 148] A is similar to E ; , for some k < n. 


3. If Aj and A: are idempotents of the same size and commute, then A) A2 is an idempotent. 

4. If A, and A, are idempotents of the same size and A; Az = AA, = 0, then A; + A) is an 
idempotent. 

5. If Ae F”*" is nilpotent, then A” = 0. 

If A is nilpotent and B is of the same size and commutes with A, then AB is nilpotent. 

7. If A; and A; are nilpotent matrices of the same size and A; A, = AA; = 0, then A; + Ap is 
nilpotent. 


a 


Examples: 


1 a ae is an idempotent i St is nilpotent. 
m2 | gg eee i eens 
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3.1 Basic Concepts 


Let V, W be vector spaces over a field F. 


Definitions: 


A linear transformation (or linear mapping) is a mapping T: V > W such that, for each u,v € V, and 
for eachc € F, T(u+v) = T(u) + T(v), and T(cu) = cT(u). 

V is called the domain of the linear transformation T: V > W. 

W is called the codomain of the linear transformation T: V > W. 

The identity transformation Iy: V — V is defined by Iy(v) = v for each v € V. Iy is also denoted 
by I. 

The zero transformation 0: V + W is defined by 0(v) = Ow for each v € V. 

A linear operator is a linear transformation T: V > V. 


Facts: 


Let T: V > W bea linear transformation. The following facts can be found in almost any elementary 
linear algebra text, including [Lan70, IV§1], [Sta69, $3.1], [Goo03, Chapter 4], and [Lay03, $1.8]. 


TOC) avi) = 0) a; T(v;), for any a; € F,v;j € V,i = 1,...,0. 
T(—v) = —T(v), for eachve V. 

The identity transformation is a linear transformation. 

The zero transformation is a linear transformation. 


ig ao" 
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3-2 Handbook of Linear Algebra 
6. If B = {v,...,V,} is a basis for V, and w,...,Wn € W, then there exists a unique T: V > W 
such that T(v;) = w; for each i. 
Examples: 


Examples | to 9 are linear transformations. 


— 


SY ea 


10. 


11. 


3.2 


x+y 


x 
T:R> > R’ where T | | y| | = 
Zz 


2x —Z 


T:V — V, defined by T(v) = —v for eachve V. 

IfAe¢ F™*", T:F" > F™, where T(v) = Av. 

T:F™*" —> F, where T(A) = trA. 

LetC(([0, 1]) be the vector space ofall continuous functions on [0, 1] intoR, andlet T :C({0,1]) ~ R 
be defined by T( f) = fy f(t)dt. 

Let V be the vector space of all functions f:IR — R that have derivatives of all orders, and 
D:V — V be defined by D( f) = f’. 


The transformation, which rotates every vector in the plane R? through an angle 0. 
x x 

The projection T onto the xy-plane of R’, i.e. T | | y =|y|. 
Zz 0 


T:R°® — R’, where T(v) = b x v, for some b € R?. 
Examples 10 and 11 are not linear transformations. 


f:R? > R’, where f ([*]) = ie 
y 


x-—y—2 
x 1 
f:1R? > R, where f ( | | = x? is not a linear transformation because f (2 A =442= 
y 


is not a linear transformation because f (0) 4 0. 


The Spaces L(V,W) and L(V,V) 


Let V, W be vector spaces over F. 


Definitions: 


L(V, W) denotes the set of all linear transformations of V into W. 
For each T,, I, € L(V, W) the sum T, + T) is defined by (T, + Th)(v) = T,(v) + T(v). 
For eachc € F, T € L(V, W) the scalar multiple c T is defined by (cT)(v) = cT(v). 
Foreach T,, Ty € L(V, V) the product T, T, is the composite mapping defined by (T, Th) (v) = Ti(Th(v)). 
T;, Ib € L(V, V) commute if T, T, = T1hT;. 
T € L(V, V) isa scalar transformation if, for some c € F, T(v) = cv for eachve V. 


Facts: 


Let T, T,, I, € L(V, W). The following facts can be found in almost any elementary linear algebra text, 
including [Fin60, $3.2], [Lan70, IV §4], [Sta69, §3.6], [SW68, §4.3], and [Goo03, Chap. 4]. 


1. 
2. 


T.t+Th €L(V,W). 
cT € L(V,W),. 
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If T,, Ty € L(V, V), then T; T € L(V, V). 

L(V, W), with sum and scalar multiplication, is a vector space over F. 

L(V, V), with sum, scalar multiplication, and composition, is a linear algebra over F. 
Let dim V = n and dim W = m. Then dim L(V, W) = mn. 

If dim V > 1, then there exist T,, Ty € L(V, V), which do not commute. 

To € L(V, V) commutes with all T € L(V, V) if and only if 7p is a scalar transformation. 


cae CAL i 


Examples: 
1. Foreach j = 1,...,mlet Tj ¢ L(F", F") be defined by T;(x) = xje;. Then ee T; is the identity 
transformation in V. 
2. Let T, and T, be the transformations that rotates every vector in R? through an angle 6; and 6, 
respectively. Then T; T; is the rotation through the angle 6; + 62. 
3. Let T; be the rotation through an angle 6 in R? and let T; be the reflection on the horizontal axis, 
that is, Ty(x, y) = (x,y). Then T; and T, do not commute. 


3.3. Matrix of a Linear Transformation 


Let V, W be nonzero finite dimensional vector spaces over F. 


Definitions: 
The linear transformation associated to a matrix A € F"*" is Ty: F” > F™ defined by T,(v) = Av. 
The matrix associated to a linear transformation T ¢ L(V, W) and relative to the ordered bases B = 
(b,,...,b,,) of V, andC of W, is the matrix ¢[T], = [[T(b,)]¢ --- [T(by)]c]. 
IfT € L(F"”, F™), then the standard matrix of T is[T] = ¢,[T]¢,, where €,, is the standard basis for F”. 
Note: If V = W and B = C, the matrix g[T]g will be denoted by [T],. 
If T € L(V, V) and B is an ordered basis for V, then the trace of T is tr T = tr [T]. 


Facts: 


Let B and C be ordered bases V and W, respectively. The following facts can be found in almost 
any elementary linear algebra text, including [Lan70, V §2], [Sta69, §3.4-3.6], [SW68, §4.3], and 
[Goo03, Chap. 4]. 


The trace of T € L(V, V) is independent of the ordered basis of V used to define it. 

For A, B € F™*", Ty = Tz if and only if A = B. 

For any T,, Ty € L(V, W), ce[ Tile = cl TJg if and only if T, = Th. 

If T € L(F", F™), then [T] = [T(e,)--- T(e,)]. 

The change-of-basis matrix from basis B to C, ¢[I],, as defined in Chapter 2.6, is the same matrix 
as the matrix of the identity transformation with respect to B and C. 

6. Let A € F”*" and let T, be the linear transformation associated to A. Then [T,] = A. 

7. IfT € L(F", F™), then Tr] = T. 
8 
9 


1 Si ee oe 


. Forany 1, T, € L(V, W), c(i + Tale = clTile t+ clhle. 
. Forany T € L(V,W),andc € F,c[cT]g =cclT]p. 
10. For any T), I € L(V,V), [ihle = (Nile (Tle. 
11. If T € L(V, W), then, for each v € V, [T(v)]e = clT]e [v]g. Furthermore ¢[T]x is the only 
matrix A such that, for each v € V, [T(v)]c = Alv]z. 
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Examples: 


1. Let T be the projection of R? onto the xy-plane of R*?. Then 
1 0 0 
[T]=]0 1 0 
0 0 0 


2. Let T be the identity in F”. Then [T]g = Ip. 
3. Let T be the rotation by 6 in R’. Then 


cos@ —sin@ 
[P= |. : 
sin@ cos 6 


4. Let D: R[x;n] — R[x;n — 1] be the derivative transformation, and let B = {1,x,...,x"},C = 
{1,x,...,x"7!}. Then 


0 1 0 0 

0 0 2 0 
clT]se = 

0 0 O n—1 


3.4 Change of Basis and Similarity 


Let V, W be nonzero finite dimensional vector spaces over F. 


Facts: 
The following facts can be found in [Gan60, III §5—6] and [Goo03, Chap. 4]. 


1. Let T € L(V, W) and let B, B’ be bases of V, C,C’ be bases of W. Then 
elTle = eWeclTlssllle. 


2. Two m x n matrices are equivalent if and only if they represent the same linear transformation 
T € L(V, W), but possibly in different bases, as in Fact 1. 
3. Any m x n matrix A of rank r is equivalent to the m x n matrix 


ot fle. 20 
i, = 
0 0 


4. Two m x n matrices are equivalent if and only if they have the same rank. 

5. Two n x n matrices are similar if and only if they represent the same linear transformation T € 
L(V, V), but possibly in different bases, i-e., if A; is similar to A), then there is T € L(V, V) and 
ordered bases 8, B2 of V such that Aj = [T]z, and conversely. 

Examples: 
1. Let T be the projection on the x-axis of R’, ie, T(x, y) = (x,0). If B = {e,,e:} andC = 


1 0 gs ye 
{e; + €2,€; — e}, then [T]g = | } [Tle = | ) and [T]e = Q™'[T]5Q with 


0 0 1/2 1/2 
cele 
o= jes 8 
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3.5 


Kernel and Range 


Let V, W be vector spaces over F and let T € L(V, W). 


Definitions: 


T is one-to-one (or injective) if v; 4 v2 implies T(v,) 4 T(v2). 
The kernel (or null space) of T is the set ker T = {v € V | T(v) = 0}. 
The nullity of T, denoted by null T, is the dimension of ker T. 


T is onto (or surjective) if, for each w € W, there exists v € V such that T(v) = w. 


The range (or image) of T is the set range T = {w € W | Av, w= T(v)}. 


The rank of T, denoted by rank T, is the dimension of range T. 


Facts: 


The following facts can be found in [Fin60, §3.3], [Lan70, IV §3], [Sta69, §3.1-3.2], and [Goo03, Chap. 4]. 


1. ker T is a subspace of V. 

2. The following statements are equivalent. 
(a) T is one-to-one. 
(b) ker T = {0}. 
(c) Each linearly independent set is mapped to a linearly independent set. 
(d) Each basis is mapped to a linearly independent set. 
(e) Some basis is mapped to a linearly independent set. 

3. range T is a subspace of W. 

4, rank T = rankc[T], for any finite nonempty ordered bases B,C. 

5. For A € F™*", ker Ty, = ker A and range T, = range A. 

6. (Dimension Theorem) Let T € L(V, W) where V has finite dimension. Then 
null T + rank T = dim V. 

7. Let T € L(V, V), where V has finite dimension, then T is one-to-one if and only if T is onto. 

8. Let T(v) = w. Then {ue V | T(uw) =w} =v-+kerT. 

9. Let V = Span{vj,...,v,}. Then range T = Span{T(v),..., T(vn)}. 

10. Let T,, T) € L(V, V). Then ker T, I > ker T; and range T, T; C range Th. 
11. Let T € L(V, V). Then 
{0} CkerT CkerT? C---CkherT*c... 
V 2 range T D range T” D-+-D range T* Dees, 
Furthermore, if, for some k, range TH = range T*, then, for eachi > 1, range TH range T*. 
If, for some k, ker T+! = ker T*, then, for each i > 1, ker Tt! = ker TK. 
Examples: 

1. Let T be the projection of R? onto the xy-plane of R*. Then ker T = {(0,0,z):z € R}; range T = 
{(x, y,0):x, y € R}; null T = 1; and rank T = 2. 

2. Let T be the linear transformation in Example 1 of Section 3.1. Then ker T = Span{[1 — 1 2]"}, 
while range T = R?. 

3. Let D € L(R[x], R[x]) be the derivative transformation, then ker D consists of all constant poly- 


nomials, while range D = R[x]. In particular, D is onto but is not one-to-one. Note that R[x] is 
not finite dimensional. 
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4. 


5. 


3.6 
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Let Ti, T, € L(F"*", F"™") where T,(A) = (A — A’), T,(A) = 5 (A+ A’), then 


ker T, = range T, = {n x n symmetric matrices}; 
ker T, = range T; = {n x n skew-symmetric matrices}; 


1 -1 
null T, = rank Ty = mnt), null T, = rank T,; = dae a 


Let T(v) = b x vas in Example 9 of Section 3.1. Then ker T = Span{b}. 


Invariant Subspaces and Projections 


Let V be a vector space over F, and let V = V, ® V for some V,, V2 subspaces of V. For each v € V, let 
v; € V; denote the (unique) vector such that v = v, + v2 (see Section 2.3). Finally, let T € L(V, V). 


Definitions: 


For i,j € {1,2},i ¢ j, the projection onto V; along V; is the operator Projy,y,;V > V defined by 
PYOjy, v, (v) = v; for each v € V (see also Chapter 5). 


The 
T is 


complementary projection of the projection projy, y, is the projection Projy,.v;- 


an idempotent if T? = T. 


A subspace Vo of V is invariant under T or T-invariant if T(Vo) C Vo. 


The 
T is 


Facts: 


fixed space of T is fix T = {ve V| T(v) = vy}. 
nilpotent if, for some k > 0, TK=0. 


The following facts can be found in [Mal63, §43—44]. 


SE ok 


Projy,y, € L(V,V). 

projy, vy, + projy,y, = I, the identity linear operator in V. 

range (projy, v,) = ker(projy, y;) = V;. 

Sum and intersection of invariant subspaces are invariant subspaces. 

If V has a nonzero subspace different from V that is invariant under T, then there exists a suitable 

An Arp Ay 
0 An Az) 

Aj, is an m-by-m block, then the subspace spanned by the first m vectors in B is a T-invariant 

subspace. 

Let T have two nonzero finite dimensional invariant subspaces V, and V2, with ordered bases B, 

and By, respectively, such that Vj @ V) = V. Let T; € L(V, Vi), Tb € L(V), V2) be the restrictions 

of T on V and V9, respectively, and let B = B, U B,. Then [T]g = [TT], © [Tx,- 


A 
ordered basis 6 of V such that [T]g = | Conversely, if [T]z = | where 


The following facts can be found in [Hoh64, 86.15; §6.20]. 


7s 
8. 


Every idempotent except the identity is singular. 
The statements 8a through 8e are equivalent. If V is finite dimensional, statement 8f is also 
equivalent to these statements. 


(a) T isan idempotent. 

(b) I — T is an idempotent. 
(c) fix T = range T. 

(d) V=kerT © fixT. 
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(e) T is the projection onto V; along V2 for some Vj, V2, with V = Vi @ Vo. 


0 


(f) There exists a basis B of V such that [T],z = Ae 


9. If T; and T, are idempotents on V and commute, then T; T) is an idempotent. 
10. If T, and T; are idempotents on V and T, T, = 7,T, = 0, then T, + T; is an idempotent. 
11. Ifdim V = nand T € L(V, V) is nilpotent, then T” = 0. 


Examples: x x 
1. Example 8 of Section 3.1, T:R*? > R*,where T | | y| | = | y| isthe projection onto Span{ej, e2} 
Zz 0 
along Span{e3}. 


2. The zero subspace is T-invariant for any T. 

3. T, and T, defined in Example 4 of Section 3.5, are the projection of F”*” onto the subspace of 
n-by-n 
symmetric matrices along the subspace of n-by-n skew-symmetric matrices, and the projection of 
F”*" onto the skew-symmetric matrices along the symmetric matrices, respectively. 

4. Let T be a nilpotent linear transformation on V. Let T? = 0 and T?-!(v) # 0. Then S = 
Span{v, T(v), T?(v),..., T?~!(v)} is a T-invariant subspace. 


3.7 Isomorphism and Nonsingularity Characterization 


Let U, V, W be vector spaces over F and let T € L(V, W). 


Definitions: 


T is invertible (or an isomorphism) if there exists a function S:W — V such that ST = Iy and 
TS = Iw. S is called the inverse of T and is denoted by T~!. 

V and W are isomorphic if there exists an isomorphism of V onto W. 

T is nonsingular if ker T = {0}; otherwise T is singular. 


Facts: 
The following facts can be found in [Fin60, §3.4], [Hoh64, $6.11], and [Lan70, IV 84]: 


. The inverse is unique. 

. T~! is a linear transformation, invertible, and (T~!)~! = T. 

. If T, € L(V, W) and T; € L(U, V), then Tj T; is invertible if and only if T, and T; are invertible. 
. If T, € L(V, W) and Ty € L(U,V), then (T1T,)“! = T, 'T,'. 

. Let T € L(V, W), and let dim V = dim W = n. The following statements are equivalent: 


ne wWN Fe 


(a) T is invertible. 
(b) T is nonsingular. 
(c) T is one-to-one. 
(d) ker T = {0}. 

(e) null T = 0. 

(f) T is onto. 


(g) range T = W. 
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(h) rank T =n. 
(i) T maps some bases of V to bases of W. 

6. If V and W are isomorphic, then dim V = dim W. 

7. Ifdim V = n > 0, then V is isomorphic to F” through g defined by g(v) = [v]g for any ordered 
basis B of V. 

8. Let dim V = n > 0, dim W = m > 0, and let B and C be ordered bases of V and W, respectively. 
Then L(V, W) and F”*" are isomorphic through g defined by g(T) = c[T]p. 


Examples: 
1. V = F[x;n] and W = F"*! are isomorphic through T € L(V, W) defined by T(37j aix') = 
[ay... dn]? 


2. If V is an infinite dimensional vector space, a nonsingular linear operator T € L(V, V) need not 
be invertible. For example, let T € L(R[x],.R[x]) be defined by T(p(x)) = xp(x). Then T is 
nonsingular but not invertible since T is not onto. For matrices, nonsingular and invertible are 
equivalent, since an n x n matrix over F is an operator on the finite dimensional vector F”. 


3.8 Linear Functionals and Annihilator 


Let V, W be vector spaces over F. 


Definitions: 
A linear functional (or linear form) on V is a linear transformation from V to F. 

The dual space of V is the vector space V* = L(V, F) of all linear functionals on V. 

If V is nonzero and finite dimensional, the dual basis of a basis 6 = {vj,...,v,} of V is the set 
B*={fi,-..5 fn} S V*, such that fi(vj) = 4;; for each i, j. 

The bidual space is the vector space V** = (V*)* = L(V*, F). 

The annihilator of a set S C Vis S° = {f € V* | f(v) =0,Vve S}. 

The transpose of T € L(V, W) is the mapping T’ € L(W*, V*) defined by setting, for each g € W*, 


T'(g):V> F 
vb g(T(v)). 


Facts: 
The following facts can be found in [Hoh64, 86.19] and [SW68, $4.4]. 


1. For eachv € V,v 4 0, there exists f € V* such that f(v) 4 0. 
2. For eachv e V define hy € L(V*, F) by setting hy( f) = f(v). Then the mapping 


g:V> Vv" 


vi> hy 


is a one-to-one linear transformation. If V is finite dimensional, g is an isomorphism of V onto V™*. 

3. S* isa subspace of V*. 

4. {0}? = V*; V? = {0}. 

5. S* = (Span{S})*. 

The following facts hold for finite dimensional vector spaces. 

6. If V is nonzero, for each basis B of V, the dual basis exists, is uniquely determined, and is a basis 
for V*. 

7. dim V = dim V*. 

8. If V is nonzero, each basis of V* is the dual basis of some basis of V. 
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9. Let B bea basis for the nonzero vector space V. For eachv € V, f € V*, f(v) = [f]lp-[v]z. 
10. If S isa subspace of V, then dim S + dim S* = dim V. 
11. If S isa subspace of V, then, by identifying V and V**, S = (S*)*. 
12. Let S), S, be subspaces of V such that S? = S$. Then S; = Sp. 
13. Any subspace of V* is the annihilator of some subspace S of V. 
14. Let S;, S, be subspaces of V. Then (S; 9 S2)* = Sf + S$ and (S; + S2)* = S7N S$. 
15. ker T? = (range T)*. 
16. rank T = rank T?. 
17. If Band C are nonempty bases of V and W, respectively, then g+[T?]e+ = (c[T]s)?. 


Examples: 


1. Let V = C[a,b] be the vector space of continuous functions yg: [a,b] — R, and let c € [a,b]. 
Then f(g) = g(c) isa linear functional on V. 

2. Let V =C[a,b], ¥ € V,and f(g) = if y(t)w(t)dt. Then f isa linear functional. 

The trace is a linear functional on F”*”. 

4. let V =F", B={Ejj:1 <i < m1 < j <n} isa basis for V. The dual basis B* consists of the 
linear functionals fjj,1 <i < m,1< j <n, defined by fjj(A) = aij. 


~ 
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4.1 Determinants 


Definitions: 
The determinant, det A, of a matrix A = [a;;] € F"*" is an element in F defined inductively: 

¢ det [a] =a. 

* Fori,j € {1,2,...,/}, theijth minor of A corresponding to a;; is defined by m;; = det A({i}, {7}). 

* The ijth cofactor of aj; is cj; = (—1)i*) mij. 

* detA= yell) Fagg; = Doi Ajj Cij fori € {1,2,...,n}. 

This method of computing the determinant of a matrix is called Laplace expansion of the determinant 

by minors along the ith row. 


The determinant ofa linear operator T: V > V ona finite dimensional vector space, V, is defined as 
det(T) = det ([T]), where BG is a basis for V. 


Facts: 


All matrices are assumed to be in F"*", unless otherwise stated. All the following facts except those with 
a specific reference can be found in [Lay03, pp. 185-213] or [Goo03, pp. 167-193]. 


1. det |“! zs = 11422 — a)2Q2}. 
421 422 
411 412 443 
2. detA = det|az, a2 a3! = 441422433 + 421413432 + 431412423 — 431413422 — 21412433 
431 432 433 
—4)1 432493. 


3. The determinant is independent of the row i used to evaluate it. 

4. (Expansion of the determinant by minors along the jth column) Let j € {1,2,...,n}. Then 
det A = SL-Di aim 4 pean ij Cij- 

5. det I, = 1. 

6. If Aisa triangular matrix, then det A = a))a22 +++ Ann. 
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7. If B isa matrix obtained from A by interchanging two rows (or columns), then det B = — det A. 
8. If B is a matrix obtained from A by multiplying one row (or column) by a nonzero constant r, 

then det B = r det A. 
9. If B is a matrix obtained from A by adding to a row (or column) a multiple of another row (or 
column), then det B = det A. 

10. If A, B, and C differ only in the rth row (or column), and the rth row (or column) of C is the sum 
of the rth rows (or columns) of A and B, then det C = det A + det B. 

11. If A is a matrix with a row (or column) of zeros, then det A = 0. 

12. If A is a matrix with two identical rows (or columns), then det A = 0. 

13. Let B be a row echelon form of A obtained by Gaussian elimination, using k row interchange 
operations and adding multiples of one row to another (see Algorithm 1 in Section 1.3). Then 
det A = (—1)* det B = (—1)' by ba9 +++ Dyn. 

14. det A? = det A. 

15. If Ae C”*", then det A* = det A. 

16. det AB = det A det B. 

17. Ifc € F, then det(c A) = c” det A. 

18. Ais nonsingular, that is Aq! exists, ifand only if det A # 0. 

19. If A is nonsingular, then det (A~!) = ae ee 

20. If S is nonsingular, then det (S~'AS) = det A. 

21. [HJ85] det A = S°, sgno @49(1)426(2) * + * Ano(n)» Where summation is over the n! permutations, o, 
of the n indices {1,2,...,n}. The weight “sgno” is 1 when o is even and —1 when o is odd. (See 
Preliminaries for more information on permutations.) 

22. Ifx,y € F”, then det(I + xy’) = +y'x. 

23. [FIS89] Let T be a linear operator on a finite dimensional vector space V. Let B and 6’ be bases 
for V. Then det(T) = det ([T],) = det ([T] 7). 

24. [FIS89] Let T be a linear operator on a finite dimensional vector space V. Then T is invertible if 
and only if det(T) # 0. 

25. [FIS89] Let T be an invertible linear operator on a finite dimensional vector space V. Then 
det(T~') = erica 

26. [FIS89] Let T and U be linear operators on a finite dimensional vector space V. Then det(TU) = 
det(T) - det(U). 

Examples: 

3-2 4 
1. Let A= | 2 5 -—6}. Expanding the determinant of A along the second column: det A = 
+3 1 5 
2- det E E. +5-det E ; — det ; 4 = 2-(—8)+5-27+4+26 = 145. 
—1 3 =2 4 
2 5 8 1 ; : : 
2. Let A = 7 =A Gel Expanding the determinant of A along the third row: det A = 
0 3 1 5 
3-2 4 —1 -2 4 —-1 3 =-2 
7- det |5 8 1] +4- det 2 8 1] +6- det 2 5 8| = 557. 
3 15 0 1 5 0 3 1 
3, LetT :IR? > R definedby T (|) eS ona fe WithB = A a | hendst (TI # 


2 -—3 i 1 l 7 eee _ 
det F ; = 15. Nowlet B’ = il? j|}-ten dete) = dt meee = 15, 
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Applications: 


1. (Cramer’s Rule) If A € F”*" is nonsingular, then the equation Ax = b, where x,b € F", has the 


S$] 
$2 
unique solutions = | , |, where s; = sh a 
Sn 
the ith column with b. 
1 1 1 
Bal X2 +++ Xy 
2 2 2 
2. [Mey00, p. 486] (Vandermonde Determinant) det | *1 *2 **' %n | = Tite ene = 23): 
x 1 tt 1 Elf xt 
3. [FB90, pp. 220-235] (Volume) Let aj, a2,...,a, be linearly independent vectors in R”. The volume, 
V, of the n-dimensional solid in R”, defined by S = (07, t1a;,0 < t < 1,i = 1,2,...,n}, is 


given by V = ,/det (A™ A), where A is the matrix whose ith column is the vector aj. 
Let m > nand T:R" — R” bea linear transformation whose standard matrix representation is 
the m x n matrix A. Let S be a region in R” of volume Vs. Then the volume of the image of S 
under the transformation T is Vrs) = \/det (A™A) - Vs. 

4, [Uhl02, pp. 247-248] (Wronskian) Let fi, fo,..., fr be n — 1 times differentiable functions of the 
real variable x. The determinant 


fix) fo(x) ss ful) 

fi(x) fi) vse fn) 
W(fir fas-- +> fa)(x) = det ’ 2 
fe) Mee ie ey flan) 


is called the Wronskian of fi, fo,...5 fn. If W(fi, frs---> fn)(x) 4 0 for some x € R, then the 
functions fi, fo,..., f, are linearly independent. 


4.2 Determinants: Advanced Results 


Definitions: 


A principal minor is the determinant of a principal submatrix. (See Section 1.2.) 
A leading principal minor is the determinant of a leading principal submatrix. 
The sum of all the k x k principal minors of A is denoted S;(A). 


The kth compound matrix of A € F”*" is the (7) x é 


matrix C;(A) whose entries are the k x k 


minors of A, usually in lexicographical order. 
The adjugate of A € F”*" is the matrix adj A = [eji] -_ [ei;] Es where c;; is the ij th-cofactor. 


The kth adjugate of A € F”*" is the « x G matrix adj (h) A whose aj; entry is the cofactor, 


in A, of the (n — k)th minor, of A, in the ijth position of the compound. 
Leta C {1,2,...,n} and A € F"*” with A[w] nonsingular. The matrix 


A/Ala] = Alo°] — Ala’, ow] A[a]7 A[a, a] 


is called the Schur complement of A[a]. 
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Facts: 
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All matrices are assumed to be in F"*", unless otherwise stated. All the following facts except those with 
a specific reference can be found in [Lay03, pp. 185-213] or [Goo03, pp. 167-193]. 


1. A(adj A) = (adj A) A = (det A) I,. 


10. 


11. 


det (adj A) = (det A)""!. 
If det A ¥ 0, then adj A is nonsingular, and (adj A)~! = (det A)~! A. 


Q@i1 412, 4430 ++ Ain 
421 422, 423, *** dn 

[Ait56] (Method of Condensation) Let A = |%31! 932 433 *** 43n|, and assume without 
Qni 4n2 4n3 +++) Ann 


loss of generality that a,; 0, otherwise a nonzero element can be brought to the (1, 1) position by 
interchanging two rows, which will change the sign of the determinant. Multiply all the rows of A 
except the first by aj. Fori = 2,3,...,n, perform the row operations: replace rowi with rowi—aj\- 


ay 412 443 Hes Qin 
O = 441422 — €21412_ 441.423 — 421413, +++ AY A2n — 421 41n 
row 1. Thus a det A = det | 9 = 411432 — 431412 411433 — 431413, +++ 1 43n — A431 AIn 
O AG. — An1@42 4114n3 — Ani 412 +++) 11 Ann — Ani @in 
41 412 Ay 413 ayy ay 
det det --» det i 
421 422 421 423 421 42n 
ay. 412 Ay a13 ayy a 
det | det -+ det ‘“ 
So, det A = hy - det 431 32 431 33 431 432 
1 
ayy 442 41 412 ayy a 
det det -++ det a 
An) Aan2 Aan an3 An ann 
[Ait56] A (adj A) = (adj “) A)A = (det A)I,. 
n—1l 


[Ait56] det (A) = det (A"), where r = peal 


[AitS6] det (A\"—®) = det(adj A), 

[HJ85] If A € F"*", B € F™*", then det (A ® B) = (det A)” (det B)”. (See Section 10.5 for the 
definition of A & B.) 

[Uhl02] For A € F"*", det A is the unique normalized, alternating, and multilinear function 
d:F"*" —» F, That is, d(I,) = 1, d(A) = —d(A’), where A’ denotes the matrix obtained from 
A, by interchanging two rows, and d is linear in each row of A, if the remaining rows of A are held 
fixed. 

[HJ85] (Cauchy-Binet) Let A € F"**, B € F**", and C = AB. Then 


det C[a, B] = 5 > det Ala, y] det Bly, B], 
Y 


where a C {1,2,...,m}, B © {1,2,...,n}, with |a| = |B] = 7,1 <r < min{m,k,n}, and the 
sum is taken over all sets y C {1,2,...,k} with |y| =r. 
[HJ85] (Schur Complement). Let A[a@] be nonsingular. Then 


det A = det A[a] det (A[a‘] — A[a’,a] Ala] 1A [a, w°]) ; 
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12. [HJ85] (Jacobi’s Theorem) Let A be nonsingular and let a, 6 C {1,2,...,n}, with || = |6|. Then 


= "i j det A[B, a] 
ge Bo] = (Ciew i+D jes A) 
det A‘ [a*, B°] = (-1) icp apa 


In particular, if ~ = B. Then det A7![a°] = det Ala | 
13. [HJ85] (Sylvester’s Identity) Leta C {1,2,...,} with |w| = k, andi, 7 € {1,2,...,n}, withi,j ¢ 


a. For A € F"*", let B = [bij] € F("-))x("—") be defined by bj; = det A [a U {i}, U {j}]. Then 


det B = (det A[a])""*"! det A. 


Examples: 
ca Da 
1. Let A = 0 = Ar S3(A) = 23 because det A[{1,2,3}] = det]O 1 0; = —3, 
0 -1 -2 -4 a ae 
1 1 0 1 0 0 
det A[{1,2,4}] = det |0 1 1] = —3, det A[{1,3,4}] = det]0 —3 2| = 16, and 
0 -1 —-4 0 —2 -4 
1 0 1 
det A[{2,3,4}] = det] 0 —3 2| = 13. From the Laplace expansion on the first column 
—-l1 -—2 -4 
and det A[{2, 3,4}] = 13, it follows that S4(A) = det A = 13. Clearly, $;(A) = tr A=—5S. 
—1 0 3 if 4 
jee Cee ee 0 -l 0 -i1 0 
1 0 1 4 3 -1 0 -—4 —3 
2. (kthcompound) Let A = ae er ee Then det A = 9,C2(A) = 1 1 3 1 4 
14 0 1 4 -1 -3 -4 —-16 
3 0 0 0 —3 
and det (C,(A)) = 729. 
= ; Bs 3 2° =3- 0 
3. (Cauchy—Binet) Let A = F = of? B=1|0 -4 4 3i),andC = AB. 
Oe eee <a ee ae 


Then det C[{2, 4}, {2, 3}] = 

det A[{2, 4}, {1, 2}] det B[{1, 2}, {2,3}] + 

det A[{2, 4}, {1, 3}] det B[{1, 3}, {2, 3}] + 

det A[{2, 4}, {2, 3}] det B[{2, 3}, {2,3}] = 12 — 44i. 


* 
4. (Schur Complement) Let A = fi C ,wherea € Cb e C™ |» andC € CPX") TEC is 
nonsingular, then det A = (a — b*C~'b) det C. Ifa # 0, then det A = a det (C — ‘bb*). 
1 -—2 0 
5. Uacobi’s Theorem) Let A = 3 4 0] anda = {2} = B. By Jacobi’s formula, det A! (2) = 
—l 0 5 
| 
5 5 0 
det Al] =f = 2. This can be readily verified by computing A~! = —< +, 0] ,and verifying 
ae 
25 «25 


ie 


det A~![{1, 3}] = 2. 
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—7 i —3 
6. (Sylvester’s Identity) Let A = | —i —2 1+4i| anda = {1}. Define B € C2? with entries 
—3 1-4 5 
bi = det A[{1, 2}] = 13, biz = det A[{1, 2}, {1, 3}] = —-7— 31i, bay = det A[{1, 3}, {1, 2}] = 
—7 + 31i, by = det A[{1,3}] = —44. Then —1582 = det B = (det A[{1}]) det A = (—7) det A, 


so det A = 226. 


4.3 Eigenvalues and Eigenvectors 


Definitions: 


Anelement A € F is an eigenvalue of a matrix A € F"™*” if there exists a nonzero vector x € F” such that 
Ax = Ax. The vector x is said to be an eigenvector of A corresponding to the eigenvalue A. A nonzero row 
vector y is a left eigenvector of A, corresponding to the eigenvalue A, ifyA = Ay. 

For A € F”*", the characteristic polynomial of A is given by p(x) = det(xI — A). 

The algebraic multiplicity, «(A), of A € o(A) is the number of times the eigenvalue occurs as a root in 
the characteristic polynomial of A. 

The spectrum of A € F"*", (A), is the multiset of all eigenvalues of A, with eigenvalue 1 appearing 
a(A) times in o (A). 

The spectral radius of A ¢ C"*" is p(A) = max{|A|:A € o(A)}. 

Let p(x) = Cx" + Cy—yx") +--+ + cox? +¢1x + €9 bea polynomial with coefficients in F. Then 
p(A) = cp A" + cpp AT +--+ 0,42 +c,A4 col. 

For A € F”*", the minimal polynomial of A, q(x), is the unique monic polynomial of least degree 
for which q4(A) = 0. 

The vector space ker(A — AI), for A € o(A), is the eigenspace of A € F”*" corresponding to A, and is 
denoted by E, (A). 

The geometric multiplicity, y (1), of an eigenvalue A is the dimension of the eigenspace E, (A). 

An eigenvalue A is simple if a(A) = 1. 

An eigenvalue A is semisimple if (A) = y (A). 

For K = Cor any other algebraically closed field, a matrix A € K"*" is nonderogatory if y (A) = 1 for 
allA € o(A), otherwise A is derogatory. Over an arbitrary field F , a matrix is nonderogatory (derogatory) 
if it is nonderogatory (derogatory) over the algebraic closure of F. 

For K = Corany other algebraically closed field, a matrix A € K"™" is nondefective if every eigenvalue 
of A is semisimple, otherwise A is defective. Over an arbitrary field F , a matrix is nondefective (defective) 
if it is nondefective (defective) over the algebraic closure of F. 

A matrix A € F"*" is diagonalizable if there exists a nonsingular matrix B € F"*", such that 
A = BDB™ for some diagonal matrix D € F"*". 


For a monic polynomial p(x) = x” + ¢,_)x"7! +--+ + c2x? + cyx + co with coefficients in F, the 
Os Ore re 0” Se6 
1 0 0 0 —C) 
n X nmatrix C(p) = 0 1:0 +++ 0  ~€2 | ig called the companion matrix of p(x). 
OG ee zeae 


Let T be a linear operator on a finite dimensional vector space, V, over a field F. An element A € F is 
an eigenvalue of T if there exists a nonzero vector v € V such that T(v) = Av. The vector v is said to be 
an eigenvector of T corresponding to the eigenvalue i. 

For a linear operator, T, on a finite dimensional vector space, V, with a basis, B, the characteristic 
polynomial of T is given by pr(x) = det([T])g. 

A linear operator T on a finite dimensional vector space, V, is diagonalizable if there exists a basis, B, 
for V such that [T]x is diagonalizable. 
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Facts: 


These facts are grouped into the following categories: Eigenvalues and Eigenvectors, Diagonalization, 
Polynomials, Other Facts. All matrices are assumed to be in F”*” unless otherwise stated. All the following 
facts, except those with a specific reference, can be found in [Mey00, pp. 489-660] or [Lay03, pp. 301-342]. 


Eigenvalues and Eigenvectors 


1. A € o(A) if and only if p4(A) = 0. 

2. For each eigenvalue A of a matrix A, 1 < y(A) < a(A). 

3. A simple eigenvalue is semisimple. 

4. For any F, |o(A)| <n. If F = C or any algebraically closed field, then |o(A)| = n. 

5. If F = C or any algebraically closed field, then det A = []j_, Ai,4; € o(A). 

6. If F = C or any algebraically closed field, then tr A = 7y_, Aj,4; € O(A). 

7. For Ae C"*", 4 € o(A) if and only if} € a (A*). 

8. For A € R"*", viewing A € C"*", 4 € o (A) ifand only ifA € o (A). 

9. If Ae C”*” is Hermitian (e.g., A € R”*” is symmetric), then A has real eigenvalues and A can be 
diagonalized. (See also Section 7.2.) 

10. Aand A? have the same eigenvalues with same algebraic multiplicities. 

11. If A = [a;;] is triangular, then o (A) = {411,422,...,4nn}. 

12. If A has all row (column) sums equal to r, then r is an eigenvalue of A. 

13. A is singular if and only if det A = 0, if and only if 0 € o(A). 

14. If A is nonsingular and A is an eigenvalue of A of algebraic multiplicity a(A), with corresponding 
eigenvector x, then A7! is an eigenvalue of A~! with algebraic multiplicity (A) and corresponding 
eigenvector x. 

15. Let Aj,A2,...,A; be distinct eigenvalues of A. For each i = 1,2,...,5 let x;),Xj2,...,Xj,, be 
linearly independent eigenvectors corresponding to A;. Then the vectors Xj1,...Xir,>X215-- +» X2ry9 

.+>Xs1>+-+5Xsr, are linearly independent. 

16. [FIS89] Let T be aa linear operator on a finite dimensional vector space over a field F , with basis 
B. Then a ¢€ F is an eigenvalue of T if and only if J is an eigenvalue of [T]p. 

17. [FIS89] Let Aj, A2,...,A; be distinct eigenvalues of the linear operator T, on a finite dimensional 
space V. For each i=1,2,...,5 let Xj), Xj2,...,Xjr, be linearly independent eigenvectors corres- 
ponding to A;. Then the vectors x11,..., Xir;)X215--+»X2r,0--+»Xs1y+-+»Xsr, are linearly independent. 

18. Let T be linear operator on a finite dimensional vector space V over a field F. Then 4 € F is an 
eigenvalue of T if and only if pr(A) = 0. 

Diagonalization 

19. [Lew91, pp. 135-136] Let A1,A2,...,A; be distinct eigenvalues of A. If A ¢ C"*", then A is 
diagonalizable if and only ifa(A;) = y(A;) fori = 1,2,...,s.If A € R”™”, then A is diagonalizable 
by a nonsingular matrix B €¢ R”™” ifand only if all the eigenvalues of A are real and @(A;) = y (Aj) 
fori = 1,2,...,s. 

20. Method for Diagonalization of A over C: This is a theoretical method using exact arithmetic and 
is undesirable in decimal arithmetic with rounding errors. See Chapter 43 for information on 
appropriate numerical methods. 

* Find the eigenvalues of A. 

° Find a basis x;1,...,Xir; for E,,(A) for each of the distinct eigenvalues A),...,A, of A. 

° Tfry+---+r, =n, thenlet B= [x1...X1,,..-Xt1-- -Xkr,]. B isinvertibleand D= B~!ABisa 
diagonal matrix, whose diagonal entries are the eigenvalues of A, in the order that corresponds 
to the order of the columns of B. Else A is not diagonalizable. 

21. A is diagonalizable if and only if A has n linearly independent eigenvectors. 
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22. A is diagonalizable if and only if |o(A)| = n and A is nondefective. 

23. If A has n distinct eigenvalues, then A is diagonalizable. 

24. A is diagonalizable if and only if q4(x) can be factored into distinct linear factors. 

25. If A is diagonalizable, then so are AT, AKkEN. 

26. If A is nonsingular and diagonalizable, then A7! is diagonalizable. 

27. If Ais an idempotent, then A is diagonalizable and 0 (A) C {0, 1}. 

28. If A is nilpotent, then o(A) = {0}. If A is nilpotent and is not the zero matrix, then A is not 
diagonalizable. 

29. [FIS89] Let T bea linear operator on a finite dimensional vector space V with a basis B. Then T is 
diagonalizable if and only if [T] is diagonalizable. 

30. [FIS89] A linear operator, T’,, on a finite dimensional vector space V is diagonalizable if and only 
if there exists a basis B = {vy,...,Vn} for V, and scalars A),...,A,, such that T(vj) = A;Vv;, for 
l<i<n. 

31. [FIS89] Ifa linear operator, T, on a vector space, V, of dimension n, has n distinct eigenvalues, 
then it is diagonalizable. 

32. [FIS89] The characteristic polynomial of a diagonalizable linear operator on a finite dimensional 
vector space can be factored into linear terms. 

Polynomials 

33. [HJ85] (Cayley-Hamilton Theorem) Let pa(x) = x” + @y,—1x""! +--+ + a,x + ag be the charac- 
teristic polynomial of A. Then pa(A) = A" + a,-; A"! +---+a,A+4 aol, = 0. 

34, [FIS89] (Cayley-Hamilton Theorem for a Linear Operator) Let pr(x) = x" + @y—1x" | + +++ + 
a,x + ao be the characteristic polynomial of a linear operator, T, on a finite dimensional vector 
space, V. Then pr(T) = T" + ay_1T" | +--+» +4,T + oI, = Ty, where Tp is the zero linear 
operator on V. 

35. par(x) = pa(x). 

36. The minimal polynomial q4(x) of a matrix A is a factor of the characteristic polynomial p(x) of 
A. 

37. If is an eigenvalue of A associated with the eigenvector x, then p(A) is an eigenvalue of the matrix 
p(A) associated with the eigenvector x, where p(x) is a polynomial with coefficients in F. 

38. If B is nonsingular, p4(x) = pp-1ag(x), therefore, A and B~! AB have the same eigenvalues. 

39. Let pa(x) = x" +ay,_\x""!+--+++4a)x +4 be the characteristic polynomial of A. If |o(A)| = 1, 
then a, = (—1)""*S,_4(Aq,...,An)) k = 0,1,...,n—1, where S(Aj,...,An) is the kth symmetric 
function of the eigenvalues of A. 

40. Let pa(x) = x" + dy_yx""! + +++ + a,x + ay be the characteristic polynomial of A. Then a, = 
(-1)""*S,_4(A),k =0,1,...,n—-1. 

41. If|o(A)| = n, then S,(A) = Sy(Aq,...5 An). 

42. IfC(p) is the companion matrix of the polynomial p(x), then p(x) = pc(p)(x) = qcrp) (x). 

43. [HJ85, p. 135] If |jo(A)| = n, A is nonderogatory, and B commutes with A, then there exists a 
polynomial f (x) of degree less than n such that B = f(A). 

Other Facts: 

44. If A is nonsingular and A is an eigenvalue of A of algebraic multiplicity w(A), with correspond- 
ing eigenvector x, then det(A)A~' is an eigenvalue of adj A with algebraic multiplicity w(A) and 
corresponding eigenvector x. 

45. [Lew91] IfA € o(A), then any nonzero column ofadj (A—AI) isan eigenvector of A corresponding 
tor. 

46. If AB = BA, then A and B have a common eigenvector. 

47. If Ae F™*" and B € F"*”, then o(AB) = o(BA) except for the zero eigenvalues. 
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48. If Ae F”*" and B € F"*", 2 € o(A), w € o(B), with corresponding eigenvectors u and v, 
respectively, then Au € o (A® B), with corresponding eigenvector u ® v. (See Section 10.5 for 
the definition of A @) B.) 


Examples: 
1. Let A= Z| . Then, viewing A € C”*”, 0 (A) = {—i, i}. That is, A has no eigenvalues over 
the reals. 
—3 7 —-1 
2. LettA= | 6 8 —2]|. Then pa(x) = (x + 6)(x — 15)? = qa(x), A, = —6, a(A1) = 1, 
72 —28 19 
y (Ay) = 1, A2 = 15, a(Az) = 2, y(A2) = 1. Also, a set of linearly independent eigenvectors is 
—1 —1 
—2|,] 1] 7. So, Ais not diagonalizable. 
4 4 
57 —21 21 
3. Let A= |] —14 22 —7/. Then pa(x) = (x + 6)(x — 15), qa(x) = (x + 6)(x — 15), 
—140 70 —55 
Ay = —6,a(A;) = 1, y(Ai1) = 1, A. = 15, a(A2) = 2, y(Az2) = 2. Also, a set of linearly 
1 1 =3 
independent eigenvectors is O},12], 1 . So, A is diagonalizable. 
2 0 10 


—54+4i 1 i 
4. Let A = 2+8i -4 2i|. Then o(A) = {—6,—3,3i}. If B = 5A? + 2A — 4], then 
20-41 -4 —-i 
o(B) = {—12, —9, —5 + 6i}. 


—2 1 0 —2 1 0 
5. Lett A = 0 —3i O| and B = 3 —1 1). B is nonsingular, so A and B-'AB = 
0 0 1 4 0 
—1+ 31 1-i i 


5+6i -—1—2i 1+ 2i| have the same eigenvalues, which are given in the diagonal of A. 
8—12i -4+4: 1-41 


3 0 
6. Lett A = E os : and B = |2 1]. Then AB = E ~N, o(AB) = 
0 3 1 7 3 
1 0 
6 —3 0 
{307 + 3V3i), (7 — 3V3i)}, BA = 4-44 sando(BA) = { 3(7 + 3V3i), $(7 — 3V/3i),0}. 
2 -l1 0 
1 1 0 —2 7 0 —2 5 0 
7. LettA=|0 1 O|}andB=J| 0 -—2 O|.ThenAB=BA=] 0 —2 O|.Aand B 
0 0 —3i 0 0 1 0 0 3 
1 
share the eigenvector x = | 0|, corresponding to the eigenvalues A = 1 of A and uw = —2 of B. 


0 
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I 1 0 0 
Ot oO 4 Boa ee 

8. LetA = 0 ae , . Then pa(x) = x*+5x°? +4x° —23x+ 13. S4(A), S3(A), and S;(A) 
0 -1l -—2 -4 


were computed in Example 1 of Section 4.2, and it is straightforward to verify that S.(A) = 4. 
Comparing these values to the characteristic polynomial, S4(A) = 13 = (—1)*13, $3(A) = 
23 = (—1)3(—23), S,(A) = (—1)?4, and S\(A) = (—1)(5). It follows that S4(A1, 2,43, 44) = 
AyAzA3A4 — 13, S3(A1, A2, 3, A4) => 23, S2(A1, 2,35 A4) => 4, and S1 (Aq, Aa; 35 A4) — ry + do + 


A3 + A4 = —5 (these values can also be verified with a computer algebra system or numerical 
software). 
0 0 -2 
9, Let p(x) = x? —7x?-3x+2,C=C(p)=]1 0 3]. Then pco(x) = x? — 7x? -3x+2= 
0 1 7 
—2 -14 —-104 0 —2 -14 
p(x). Also, pc(C) = -C? +7C7+3C-21=—-]|] 3 19 142/4+7]/0 3. 19 
7 52 383 1 7 52 
0 0 -2 1 0 0 0 0 0 
+3/1 0 3} —2/10 1 0} =]|0 O O 
0 1 7 0 0 1 0 0 0 
Applications: 
1. (Markov Chains) (See also Chapter 54 for more information.) A Markov Chain describes a 


2 


process in which a system can be in any one of n states: s),52,...,5,. The probability of en- 
tering state s; depends only on the state previously occupied by the system. The transition 
probability of entering state j, given that the system is in state i, is denoted by p;;. The tran- 
sition matrix is the matrix P = [pjj;]; its rows have sum 1. A (row or column) vector is 
a probability vector if its entries are nonnegative and sum to 1. The probabilty row vector 
mh) = (nf 7, pe a), k > 0, is called the state vector of the system at time k if its ith 
entry is the probability that the system is in state s; at time k. In particular, when k = 0, the 
state vector is called the initial state vector and its ith entry is the probability that the system 
begins at state s;. It follows from probability theory that w+) = a P, and thus inductively 
that w = © PX, If the entries of some power of P are all positive, then P is said to be 

My My +t Ty 

My My +s Ty 
regular. If P is a regular transition matrix, then asn — 00, P" > |. |, . |. The 


Il IT sae Ty 


row vector 7 = (71,71,...,H») is called the steady state vector, 7 is a probability vector, and 

as n — 00, wm) —> 7. The vector 7 is the unique probability row vector with the property 

that wP = 7. That is, w is the unique probability row vector that is a left eigenvector of P for 

eigenvalue 1. 

(Differential Equations) [Mey00, pp. 541-546] Consider the system of linear differential equations 
Xp = ayx + ayx. +o... 4H AinXn 


Xy = anx, + aynx. +... 4+ ArnXn 
, where each of the unknowns x), X2,...5Xn 


xX, = AnX, + AnX. + ... 4+ AnnXy 
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is a differentiable function of the real variable t. This system of linear differential equations can 


x1 (t) x; (t) 
= Ae x2(t) x;(t) 

be written in matrix form as x’ = Ax, where A = [ajj],x = . |,andx = . |. fA 
Xn(t) x;, (t) 


is diagonalizable, there exists a nonsingular matrix B (the columns of the matrix B are linearly 
independent eigenvectors of A), such that B-!AB = Disa diagonal matrix, so x’ = BDB —lx, or 


B~!x’ = DB~!x. Letu = B~!x. The linear system of differential equations u’ = Du has solution 
kye*" 
koe! 

u= ‘ , where Aj,A2,...,Ay are the eigenvalues of A. It follows that x = Bu. (See also 
poke 


Chapter 55.) 
3. (Dynamical Systems) [Lay03, pp. 315-316] Consider the dynamical system given by uz,; = Aux, 


ay 
a2 

where A = [ajj],uo = | . | . If Ais diagonalizable, there exist n linearly independent eigenvectors, 
an 

X15 X2,...5X;, Of A. The vector up can then be written as a linear combination of the eigenvectors, 


that is, up = cyxX; + C2X) +--+ + C,X,. Then uy = Aug = A(C,X, + 62%. +--+ + CyX,) = 
CyAyXy +02A9%) ++ ++ + CnAnXy. Inductively, uzy) = Auy = cy Akt x, Siegal ey +.-- +c, At x, 
Thus, the long-term behavior of the dynamical system can be studied using the eigenvalues of the 
matrix A. (See also Chapter 56.) 
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5.1 Inner Product Spaces 


Definitions: 


Let V be a vector space over the field F, where F = R or F = C. An inner product on V is a function 
(5): V x V — F such that for all u,v, w € V anda,b € F, the following hold: 


* (v,v) = Oand (v,v) = 0 if and only ifv = 0. 
* (au + by,w) = a(u,w) + b(v,w). 
¢ For F = R: (u,v) = (v,u); For F = C: (u,v) = (v,u) (where bar denotes complex conjugation). 


A real (or complex) inner product space is a vector space V over R (or C), together with an inner 
product defined on it. 

In an inner product space V, the norm, or length, of a vector v € V is ||v|| = \/(v,v). 

A vector v € V isa unit vector if ||v|| = 1. 

The angle between two nonzero vectors u and v in a real inner product space is the real number 6, 
0 < 6 <7, such that (u,v) = ||ull||v|| cos@. See the Cauchy—Schwarz inequality (Fact 9 below). 

Let V be an inner product space. The distance between two vectors u and vis d(u, v) = ||u — v||. 
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A Hermitian matrix A is positive definite if x* Ax > 0 for all nonzero x € C”. (See Chapter 8 for more 
information on positive definite matrices.) 


Facts: 
All the following facts except those with a specific reference can be found in [Rom92, pp. 157-164]. 


1. The vector space R” is an inner product space under the standard inner product, or dot product, 
defined by 


n 
(u,v) = wv= Se UjV;- 
i=l 


This inner product space is often called n—dimensional Euclidean space. 
2. The vector space C” is an inner product space under the standard inner product, defined by 


n 
(u,v) =Viu= a” UjV}- 
i=l 


This inner product space is often called n-dimensional unitary space. 

3. [HJ85, p. 410] In R”, a function (-,-): R” x R"” — R isan inner product if and only if there exists 
a real symmetric positive definite matrix G such that (u,v) = u! Gy, for all u,v € R”. 

4. [HJ85, p. 410] In C”, a function (-,-):C” x C” — C is an inner product if and only if there exists 
a Hermitian positive definite matrix H such that (u,v) = v* Hu, for all u,v € C”. 


5. Let I? be the vector space of all infinite complex sequences v = (v,) with the property that 
oe) 


be |v,|° < oo. Then /? is an inner product space under the inner product 


n=1 


CO 
(u,v) = S UnVn- 
n=1 


6. The vector space C [a, b] of all continuous real-valued functions on the closed interval [a, b] is an 
inner product space under the inner product 


b 
(fg) = i ener es 


7. If V is an inner product space and (u, w) = (v, w) for all w € V, then u = v. 

8. The inner product on an inner product space V, when restricted to vectors in a subspace S of V, 
is an inner product on S. 

9. Let V be an inner product space. Then the norm function || - || on V has the following basic 
properties for all u,v € V: 


* ||v|| = 0 and ||v|| = 0 if and only ifv = 0. 


|av|| = |a|||vl|, for alla € F. 


° (The triangle inequality) ||u + v|| < ||ul| + ||v|| with equality if and only if v = au, for some 
aeéF. 


(The Cauchy—Schwarz inequality) | (u, v)| < ||ull||v|| with equality ifand only ifv = au, for some 
aecF. 


{lull — IIvIll < lu — vil. 
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* (The parallelogram law) |[u + v||? + [lu — vil? = 2|ull? + 2\IvIl?. 


¢ (Polarization identities) 


Ju+ vi? — Ju—vil?, if F =R. 
4(u,v) = 
ju + vi]? — |ju — vil? + iJJu + ivi? — illu — ivi’, if F =C. 


Examples: 


1. Let R* be the Euclidean space with the inner product (u,v) = u’v. Let x = [1,2,3,4]’ € R* and 
y = [3,-1,0,2]7 € R* be two vectors. Then 


* (x,y) = 9, ||xl] = /30, and |ly|| = /14. 
* The distance between x and y is d(x, y) = ||x — y|| = J26. 


9 9 
* The angle between x and y is 9 = arccos = arccos ~ 1.116 radians. 


30/14 2/105 


1 2 F é : 
2. (u,v) = uy vy + 2 V2 + 2uv) + 6inV2 = uw ; 6 v is an inner product on R’, as the matrix 


1 ae ae : i : 
G= | , ; is symmetric positive definite. 
1 


3. Let C[—1, 1] be the vector space with the inner product ( f, g) = i f(x)g(x)dx and let f(x) = 1 
-1 


1 1 
and g(x) = x*betwo functionsin C[—1, 1]. Then (f,g) = ‘| d= f= / ldx = 2, 
| 1 


1 
and (g,g) = / x‘dx = 2/5. The angle between f and g is arccos(/5/3) © 0.730 radians. 
-1 
4, [Mey00, p. 286] (A, B) = tr(AB*) is an inner product on C™™". 


5.2 Orthogonality 


Definitions: 


Let V be an inner product space. Two vectors u,v € V are orthogonal if (u,v) = 0, and this is denoted 
byu Lv. 

A subset S of an inner product space V is an orthogonal set if u _L v, for all u,v € S such that u ¥ v. 

A subset S of an inner product space V is an orthonormal set if S is an orthogonal set and each v € S 
is a unit vector. 

Two subsets S and W of an inner product space V are orthogonal if u _L v, for all u € S andve W, 
and this is denoted by S L W. 

The orthogonal complement of a subset S of an inner product space V is St = {w € V|(w,v) = 
0 forall ve S}. 

A complete orthonormal set M in an inner product space V is an orthonormal set of vectors in V such 
that for v € V,v L M implies that v = 0. 

An orthogonal basis for an inner product space V is an orthogonal set that is also a basis for V. 

An orthonormal basis for V is an orthonormal set that is also a basis for V. 

A matrix U is unitary if U*U = I. 

A real matrix Q is orthogonal if Q'Q=l. 
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Facts: 
1. 


10. 
11. 


12. 


13. 


14. 


15. 


16. 


Handbook of Linear Algebra 


[Mey00, p. 298] An orthogonal set of nonzero vectors is linearly independent. An orthonormal set 
of vectors is linearly independent. 


. [Rom92, p. 164] If S is a subset of an inner product space V, then S+ is a subspace of V. Moreover, 
p Pp p p 


if S is a subspace of V, then SM S+ = {0}. 


. [Mey00, p. 409] In an inner product space V, {0} = V and V+ = {0}. 
. [Rom92, p. 168] If S is a finite dimensional subspace of an inner product space V, then for any 


ve V, 
* There are unique vectors s € S andt € S+ such that v= s+ t. This implies V = S @ St. 


* There is a unique linear operator P such that P(v) = s. 


. [Mey00, p. 404] If S is a subspace of an n—dimensional inner product space V, then 


e(Stit=s. 
* dim(S+) = n— dim(S). 


. [Rom92, p. 174] If S is a subspace of an infinite dimensional inner product space, then S C (S+)+, 


but the two sets need not be equal. 


. [Rom92, p. 166] An orthonormal basis is a complete orthonormal set. 
. [Rom92, p. 166] Ina finite-dimensional inner product space, a complete orthonormal set is a basis. 
. [Rom92, p. 165] In an infinite-dimensional inner product space, a complete orthonormal set may 


not be a basis. 

[Rom92, p. 166] Every finite-dimensional inner product space has an orthonormal basis. 

[Mey00, p. 299] Let 6 = {uj, w,...,u,} be an orthonormal basis for V. Every vector v € V can 
be uniquely expressed as 


n 


v= So (v, ui) uj. 


i=1 


The expression on the right is called the Fourier expansion of v with respect to B and the scalars 
(v, u;) are called the Fourier coefficients. 
[Mey00, p. 305] (Pythagorean Theorem) If (vie, is an orthogonal set of vectors in V, then 


Sev ell? 
[Rom92, p. 167] (Bessel’s Inequality) If {uj}E, is an orthonormal set of vectors in V, then 


Iv? = Sha Kv, ws) P. 
[Mey00, p. 305] (Parseval’s Identity) Let B = {u,, u2,...,u,} be an orthonormal basis for V. Then 


for each v € V, |v||? = = I(v, u;)|°. 
i=l 


[Mey00, p. 405] Let A € F”*", where F = Ror C. Then 

* ker(A)t = range(A*), range(A)+ = ker(A*). 

* F™ = range(A) @ range(A)+ = range(A) @ ker(A*). 

* F" = ker(A) @ ker(A)+ = ker(A) @ range(A*). 

[Mey00, p. 321] (See also Section 7.1.) The following statements for a real matrix Q € R"*” are 
equivalent: 

* Qis orthogonal. 

* Q has orthonormal columns. 

* Q has orthonormal rows. 

* QQ!’ = I, where J is the identity matrix of order n. 


* Forallv € R’", || Qv|| = |lvil. 
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17. 


[Mey00, p. 321] (See also Section 7.1.) The following statements for a complex matrix U € C”*" 
are equivalent: 


* U is unitary. 

* U has orthonormal columns. 

* U has orthonormal rows. 

* UU* = I, where I is the identity matrix of order n. 


* Forallv € C’, ||Uv|| = |v. 


Examples: 
1 
1. Let C{—1, 1] be the vector space with the inner product ( f, g) = / f(x)g(x)dx and let f(x) = 1 
-1 


5.3 


1 
and g(x) = x be two functions in C[—1, 1]. Then (f,g) = xdx = 0. Thus, f 1 g. 


The standard basis {e), e2,...,e,} is an orthonormal basis for. the unitary space C”. 
If (vj, V2,- ++, Vn} is an orthogonal basis for C” and S = span {vj}, v2,---, vg} (1 < k < n—1), then 
SS span {Vi415°°+5 Vn}. 


The vectorsv, = [2,2,1]’,v. = [1,—1,0]’,andv3 = [-1, -1, 4]? are mutually orthogonal. They 
can be normalized to u; = v,/||v; || = [2/3, 2/3, 1/3]"; w = vo/llvoll| = (1/2, -1/72,0]", and 
uw = ¥3/Ilv3l| = [—/2/6, —V2/6, 2/2/3]". The set B = {uj, uy, u3} forms an orthonormal 
basis for the Euclidean space R?. 


° Ifv = [v1, v2, v3]? € R°, then v = (v, uy) uy, + (v, uy) uy + (v, us) us, that is, 


eu tt ee Vs ~via v2 V1 — V2 + 43 


u, 4 ut+ 
3 rT ats 502 


U3. 


* The matrix Q = [uy, up, us] € R**? is an orthogonal matrix. 


. Let S be the subspace of C? spanned by the vectors u = [i, 1,1]? and v = [1,i,1]7. Then the 


orthogonal complement of S is 


S+ = {wlw = a[1,1,-1+i]", where a € C}. 


. Consider the inner product space /? from Fact 5 in Section 5.1. Let € = {e;|i = 1,2,...}, where 


e; has a 1 on ith place and 0s elsewhere. It is clear that € is an orthonormal set. If v = (v,) L E, 
then for each n, v, = (v,e,) = 0. This implies v = 0. Therefore, € is a complete orthonormal set. 
However, € is not a basis for /* as S = span{€} # I’. Further, St = {0}. Thus, (S+)+ =P Z S 
andi?#S@S-. 


Adjoints of Linear Operators on Inner Product Spaces 


Let V be a finite dimensional (real or complex) inner product space and let T be a linear operator on V. 


Definitions: 


A linear operator T* on V is called the adjoint of T if (T(u),v) = (u, T*(v)) for allu,v € V. 
The linear operator T is self-adjoint, or Hermitian, if T = T*; T is unitary if T*T = Iy. 
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Facts: 


Handbook of Linear Algebra 


The following facts can be found in [HK71]. 


1. Let f bea linear functional on V. Then there exists a unique v € V such that f(w) = (w,v) for 
allwe V. 
2. The adjoint T* of T exists and is unique. 
3. Let B = (uy, u,...,u,) be an ordered, orthonormal basis of V. Let A = [T]. Then 
aij = (T(uj), uj), 1,7 =1,2,...,0. 
Moreover, [T*]g = A*, the Hermitian adjoint of A. 
4, (Properties of the adjoint operator) 
(a) (T*)* = T for every linear operator T on V. 
(b) (aT)* = aT™ for every linear operator T on V and everya € F. 
(c) (T + T,)* = T* + T,* for every linear operators T, T; on V. 
(d) (TT,)* = T,*T* for every linear operators T, T; on V. 
5. Let B be an ordered orthonormal basis of V and let A = [T]. Then 
(a) T is self-adjoint if and only if A is a Hermitian matrix. 
(b) T is unitary if and only if A is a unitary matrix. 
Examples: 
1. Consider the space R* equipped with the standard inner product and let f(w) = 3w, — 2w3. 
Then with v = [3,0, —2]7, f(w) = (w,v). 
x 
2. Consider the space R* equipped with the standard inner product. Let v = |y] and T(v) = 
Zz 
2x+y 2 0 2 0 1 2x + Z 
y —3z |.Then[T] = |0 —3],so[T]* = }1 1 1},and T*(v) = |x+y+z|]. 
xtytz 1 1 O62. 1 —3y+z 
3. Consider the space C”*” equipped with the inner product in Example 4 of section 5.1. Let A, B € 
C"*" and let T be the linear operator on C”*" defined by T(X) = AX + XB, X € C”*". Then 
T*(X) = A*X4+ XB*, XE C™". 
4. Let V be an inner product space and let T be a linear operator on V. For a fixed u € V, f(w) = 


5.4 


(T(w), u) is a linear functional. By Fact 1, there is a unique vector v such that f(w) = (w, v). Then 
T*(u) =v. 


Orthogonal Projection 


Definitions: 


Let S bea finite-dimensional subspace ofan inner product space V. Then according to Fact 4 in Section 5.2, 
each v € V can be written uniquely as v = s + t, wheres € S andt € S". The vector s is called the 
orthogonal projection of v onto S and is often written as Proj,v, where the linear operator Proj, is called 
the orthogonal projection onto S along St. When V = C” or V = R" with the standard inner product, 
the linear operator Proj, is often identified with its standard matrix [Proj,] and Proj, is used to denote 
both the operator and the matrix. 
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Facts: 

1. An orthogonal projection is a projection (as defined in Section 3.6). 

2. [Mey00, p. 433] Suppose that P is a projection. The following statements are equivalent: 
* P isan orthogonal projection. 

° p* =P. 
* range(P) L ker(P). 
3. [Mey00, p. 430] If S is a subspace of a finite dimensional inner product space V, then 
Projsi = I — Projs. 

4, [Mey00, p. 430] Let S be a p—dimensional subspace of the standard inner product space C”, and 
let the columns of matrices M € C”*? and N € C"*‘"~P) be bases for S$ and S+, respectively. Then 
the orthogonal projections onto S and S+ are 

Projs = M(M*M)7'M* and Projo. = N(N*N)7'N*. 
If M and N contain orthonormal bases for S and S+, then Projy = MM* and Projo. = NN*. 

5. [Lay03, p. 399] If {u;,..., up} is an orthonormal basis for a subspace S of C”, then for any v € C”, 

Proj.v = (ujv)u; +--+ + (u,v)Up. 
6. [TB97, p. 46] Let v € C” be a nonzero vector. Then 
vv" 
* Proj, = —~ is the orthogonal projection onto the line L = span{v}. 
viv 
vv* 
* Proj, = J — —~ is the orthogonal projection onto BS: 
viv 

7. [Mey00, p. 435] (The Best Approximation Theorem) Let S be a finite dimensional subspace of an 
inner product space V and let b be a vector in V. Then Proj,b is the unique vector in S that is 
closest to b in the sense that 

min ||/b — s|| = ||b — Proj,b]]. 
seS 
The vector Projcb is called the best approximation to b by the elements of S. 
Examples: 

1. Generally, an orthogonal projection P € C”*” is not a unitary matrix. 

2. Let {v1,V2,--+,V,} be an orthogonal basis for R” and let S the subspace of R” spanned by 
{v],---, vg}, where 1 < k < n—1. Thenw = civ, + cov. +--+ + C,V, € R" can be writ- 
ten as w =s+t, wheres = cyvj + +--+ cyve € Sandt = cyyyVeg) He? HF enVy € St. 

3. Let uw) = [2/3,2/3,1/3]", uw = [1/3, -2/3,2/3]", and x = [2,3,5]". Then {uj, uy} is an or- 


thonormal basis for the subspace S = span {uj, uy} of R>. 


* The orthogonal projection of x onto S is 


Proj;x = (uf) xu + (u?x)u, =| AsS | P3 


* The orthogonal projection of x onto S+ is y = x — Projgx = [—2, 1,2]7. 


* The vector in S that is closest to x is Proj,x = [4, 2, 3] f 
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° Let M = [uj, uy]. Then the orthogonal projection onto S is 


ale 2 A 
Proj = MM’ =-|2 8 —2 
a ee 


* The orthogonal projection of any v € R? onto S can be computed by Projpv = MM'Vv. In 


particular, MM’x = [4,2,3]?. 
4. Let w, = [1,1,0]! and w, = [1,0, 1]. Consider the subspace W = span{w, w2} of R*. Define 


,0, 
1 
0 

1 2 
1 


1 
: 2s: 
the matrix M = [w),w2] = | 1 .Then MM = | } 
0 


* The orthogonal projection onto W is Projy = M(M'M)-!M? = 
1 1 = 
ae ; l 
1 2 
0 1 


* The orthogonal projection of any v € R? onto W can be computed by Projyv. For v = [1,2,3]7, 
Proj v= Proj yl, 2,3) =(7/3,2/3,5 81". 


1 -l 2 


5.5 Gram—Schmidt Orthogonalization and QR Factorization 


Definitions: 
Let {a),a,...,a,} be a basis for a subspace S of an inner product space V. An orthonormal basis 
{uj,U),...,u,} for S can be constructed using the following Gram—Schmidt orthogonalization process: 
k-1 
ay a — >) (aks Ui) U 
l= uu, = a , for k=2,...,n. 
llai ll Ila, — dof (aK Us) Us Il 


A reduced QR factorization of A € C”*" (m > n) isa factorization A = OR, where Q € C”*" has 
orthonormal columns and R € C"*" is an upper triangular matrix. 

A QR factorization of A € C”*” (m > n) is a factorization A = QR, where Q € C”*” is a unitary 
matrix and R € C”™” is an upper triangular matrix with the last m — n rows of R being zero. 


Facts: 


1. [TB97, p. 51] Each A € C”*" (m > n) has a full QR factorization A = QR. If A € R”*", then 
both Q and R may be taken to be real. 

2. [TB97, p. 52] Each A € C”*" (m > n) has a reduced QR factorization A = OR. If A € R™", 
then both Q and R may be taken to be real. 

3. [TB97, p.52] Each A € C"*" (m > n) of full rank has a unique reduced QR factorization A = OR, 
where Q € C”*" and R € C"*" with real r;; > 0. 

4, [TB97, p. 48] The orthonormal basis {u),uw,...,u,} generated via the Gram—Schmidt ortho- 
gonalization process has the property 


Span({uj,U2,...,u}) = Span({aj,a2,...,ax}), 


fork = 1,2,...,n. 
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5. 


6. 


[TB97, p. 51] 


Algorithm 1: Classical Gram—Schmidt Orthogonalization: 


input: a basis {a),a2,...,a,} for a subspace S 
output: an orthonormal basis {u,, u,...,u,} for S 
forj =1:n 

uj = aj 

fori=1:j-—1 


rj = (aj, uj) 
Uj = Uj << rjjUj 


end 
rij c= lull 
uj = uj/T jj 
end 
[TB97, p. 58] 


Algorithm 2: Modified Gram—Schmidt Orthogonalization 


input: a basis {a),a2,...,a,} for a subspace S 
output: an orthonormal basis {u;,u,...,u,} for S 
w; :=aj,,i=1:n 
fori=1:n 

Tit = [will 

Uj = Wi /Tii 


forj=i+l:n 
ij = (wj, Uj) 
Wj = Wj — rij Ui 
end 
end 


[Mey00, p. 315] If exact arithmetic is used, then Algorithms 1 and 2 generate the same ortho- 
normal basis {u;, U2,...,Uu,} and the same rjj;, for j > i. 

[GV96, pp. 230-232] If A = [aj,a2,...,a,] € C”*”" (m > n) is of full rank n, then the classic 
or modified Gram—Schmidt process leads to a reduced QR factorization A = OR, with Q = 
[uj,U,...,U,] and Rij = rij, for j =i, and Rij = 0, for j <i. 

[GV96, p. 232] The costs of Algorithm 1 and Algorithm 2 are both 2mn? flops when applied to 
compute a reduced QR factorization of a matrix A € R”*". 


. [Mey00, p. 317 and p. 349] For the QR factorization, Algorithm 1 and Algorithm 2 are not numer- 


ically stable. However, Algorithm 2 often yields better numerical results than Algorithm 1. 


. [Mey00, p. 349] Algorithm 2 is numerically stable when it is used to solve least squares problems. 
. (Numerically stable algorithms for computing the QR factorization using Householder reflections 


and Givens rotations are given in Chapter 38.) 


. [TB97, p. 54] (See also Chapter 38.) If A = QR isa QR factorization of the rank n matrix A € C”"*", 


then the linear system Ax = b can be solved as follows: 
* Compute the factorization A = QR. 
* Compute the vector c = Q*b. 


* Solve Rx = c by performing back substitution. 
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Examples: 


— 
N 


1. Consider the matrix A= |2 0]. 
0 2 


¢ Ahasa (full) QR factorization A = QR: 


u this, 252 2 
12) [wa sa 3] [> x 
ae |e ee 7 6 
2 0l=|% -s= G]] 0 & 
0 2 ae ae eS a) 
* Ahasa reduced QR factorization A = QR: 
as B ee 
2 of =|4 -Z2 - 
a 3/5 0 a : 
Js 5 
0 2 0 > 
3 Lo 2 
P : 3. —4 : F ; : 
2. Consider the matrix A = a noe ale Using the classic or modified Gram—Schmidt process 
3 1 0 
gives the following reduced QR factorization: 
d; ——— 
3 Li oe 2 2 2 2 
aD, ogc Peep) sO seed: ed 
3. —-4 1] 3 2 2| |o 5-1 
3 -4 -1} |i ~1 _1 0 0 , 
2 2 2 
3 1 0 1 1 1 
2 2 2 


5.6 Singular Value Decomposition 


Definitions: 


A singular value decomposition (SVD) of a matrix A € C”*” is a factorization 
A=UZXV", X= diag(o},02,...,0)) € R"*”, p = min{m, n}, 


where o; > 02 >... > 6p => Oand both U = [u),uy,...,u,] € C"” and V = [v1,v2,..., Vn] € C"*" 
are unitary. The diagonal entries of & are called the singular values of A. The columns of U are called left 
singular vectors of A and the columns of V are called right singular vectors of A. 

Let Ae C”*" with rankr < p = min{m, n}. A reduced singular value decomposition (reduced SVD) 
of A is a factorization 


A=USV*, Y= diag(o),02,...,0,) € R’*’, 


where 0; > 0) >... > 0, > Oand the columns of U = [u),u,...,u,] € C”*" and the columns of 


A 


V = [v,,v2,...,v,] € C””” are both orthonormal. 
(See §8.4 and $3.7 for more information on singular value decomposition.) 
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Facts: 


All the following facts except those with a specific reference can be found in [TB97, pp. 25-37]. 


1. 


G0 


10. 


11. 


12. 
13. 


Every A € C”*" has a singular value decomposition A = UXV*. If A € R”*", then U and V 
may be taken to be real. 

The singular values of a matrix are uniquely determined. 

If A € C”*" has a singular value decomposition A = UX V*, then 


Av; = ojUj, A uj = OjVj, uj Av; = 0j; 


for j = 1,2,..., p = min{m, n}. 

If UXV* is a singular value decomposition of A, then VE? U* is a singular value decomposition 
of A*. 

If A € C”*" has r nonzero singular values, then 


¢ rank(A) =r. 
“A= So ojujy;. 
j=l 


* ker(A) = span{v,41,..-5Vn}. 
* range(A) = span{u,,...,u,}. 


Any A € C”™*" of rankr < p = min{m, n} has a reduced singular value decomposition, 
A=UdV*, S= diag(o),02,...,0,) € R’*’, 


where 0} > 0) >--- > 0, > Oand the columns of U = [u,,u,...,u,] € C”*" and the columns 
of V = [v1,v2,...5v,] € C”** are both orthonormal. If A € R”*”, then U and V may be taken 
to be real. 

If rank(A) = r, then A hasr nonzero singular values. 

The nonzero singular values of A are the square roots of the nonzero eigenvalues of A* A or AA”. 
[HJ85, p. 414] If UXV* is a singular value decomposition of A, then the columns of V are 
eigenvectors of A* A; the columns of U are eigenvectors of AA”. 

[HJ85, p. 418] Let A € C”*" and p = min{m, n}. Define 


G= 0 ‘ A E Cita) x (mn) 
A 0 
If the singular values of A are o1,...,0p, then the eigenvalues of G are 0),...,05,—01,...,—Op 
and additional |n — m| zeros. 
If A € C"*" is Hermitian with eigenvalues 41,42,---+,An, then the singular values of A are 
lAql|; |A2|,° why [Anl- 


For A € C"*", |det A] = 0102 -++ Oy. 

[Aut15; Sch07] (Eckart-Young Low Rank Approximation Theorem) 

Let A= UX V* bean SVD of A € C”™" andr = rank(A). Fork < r, define Ay = ae oj UjV}- 
Then 


° ||A— Agll2= min ||A— Bll, = o¢415 
rank(B)<k 


° ||A— Agile = min ||A— Bllp = 
rank(B)<k 


m n 
y s mi are the 2-norm and Frobenius norm of 
i=1 j=l 


where || M||. = mae, || Mx||2 and || M||- = 
ie 


x|| 


matrix M, respectively. (See Chapter 37 for more information on matrix norms.) 
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Examples: 
ae 1 2 0 
Consider the matrices A= |2 0] andB= A! = 6 ; . 
0 2 


1. The eigenvalues of ATA = are 9 and 4. So, the singular values of A are 3 and 2. 


aT, 2 
2. Normalized eigenvectors for A? A are vy; = | and v2 = | | ‘ 


v5 v5 
8 0 
3. uw) = Ay) = ue and u, = 5 Av) = z . Application of the Gram—Schmidt process to 

4 seeks 

35 3 
2 
3 
Uj, Up, and e; produces us = | —} 
2 
3 


4. Ahas the singular value decomposition A = UV", where 


+ ‘lees 20 2/5 3 0 ete. 3 
U=—=|2 6 ~5],2=|0 2), v= | | 
2 -1 

3V5 14 3 as 0 0 5 


II 
ran 
M 
<=> 
4 
< 
> 
oO 
o 


5. Ahas the reduced singular value decomposition A 


5 0 
‘ 1 “ 3 0 nN 1 }1 2 
=——|2 6|,%= ,V=— 
U 3/5 0 2 |: = 
4 —3 
6. B has the singular value decomposition B = Ug Zz Vz , where 
5 0) 2/5 
1 /1 2 3 0 0 
Uz = Va = — }2=| , Vp =U,g=—=l2 6 —-V5 
2 —-1 2 0 
ve ; OU Na 5. 29s 


(U, = U and Vy = V for A were given in Example 4.) 


5.7 Pseudo-Inverse 


Definitions: 


A Moore-Penrose pseudo-inverse of a matrix A € C’*" isa matrix At ¢ C"*" that satisfies the following 
four Penrose conditions: 


AA'A= A; A'AAT= Al; (AAT)* = AAT; (ATA)* = ATA, 
Facts: 
All the following facts except those with a specific reference can be found in [Gra83, pp. 105-141]. 


1. Every A € C”*" has a unique pseudo-inverse A‘. If A € R”*”, then A? is real. 
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2: 


15. 


16. 
IVE 
18. 


19, 


[LH95, p. 38] If A € C”*" ofrankr < min{m,n}hasanSVD A = UXV*, then its pseudo-inverse 
is Ai = VX'U*, where 


x! = diag(1/o;,...,1/0,,0,...,0) € R™™. 


0, = Onm and hy — Jum where Om, € C”*" is the all Os matrix and J inn € C™”*” is the all 1s 
matrix. 

(Al)* = (A*)I; (AT =A. 

If A is a nonsingular square matrix, then A‘=A7l, 

If U has orthonormal columns or orthonormal rows, then Ut = U*. 

If A= A* and A = A’, then Al = A. 

Al = A’ if and only if A* A is idempotent. 

If A= A*, then AAT = ATA. 

If U € C”*" is of rank n and satisfies Ut = U*, then U has orthonormal columns. 


. IU € C”™™ and V € C"*" are unitary matrices, then (UAV)! = V*ATU*. 
. If Ae C”™" (m > n) has full rank n, then At = (A*A)7! A*. 

. If Ae C”™*" (m < n) has full rank m, then At = A*(AA*)7!. 

. Let Ae C”*", Then 


° AltA, AAt, I, — AtA, and I, — AAt are orthogonal projections. 

* rank(A) = rank(A') = rank(AA') = rank(A! A). 

* rank(I,, — Al A) = n — rank(A). 

* rank(I, — AA‘) = m — rank(A). 

If A = A; + Ap +--+ + Ay, Af Aj = 0, and Aj Aj = 0, for all i,j = 1,---,k,i # j, then 
At=Al+aAl+---4+ al. 

If Ais an m x r matrix of rankr and B is anr x n matrix of rankr, then (AB)! = BI At, 
(A*A)t = AT(A*)I; (AA*)T = (A*)TAT. 

[Gre66] Each one of the following conditions is necessary and sufficient for (AB )i = BiAl: 

* range(B B* A*) C range(A*) and range(A* AB) C range(B). 

+ Ai ABB* and A* AB B' are both Hermitian matrices. 

* Al ABB*A* = BB* A* and BBtA*AB = A* AB. 

* A'ABB*A*ABBt = BB*A*A. 

* ATAB = B(AB)'AB and BBtA* = A*AB(AB)!. 

Let A € C”*" and b € C”. Then the system of equations Ax = b is consistent if and only if 


AA'‘b = b. Moreover, if Ax = b is consistent, then any solution to the system can be expressed as 
x= Aib+ (I, — At A)y for some y € C”. 


Examples: 


1, 


2: 


1 2 
: ‘ ; 2 8 —2 
The pseudo-inverse of the matrix A= |2 0] is A’ = i | | ; 
0 2 


(AB)! = B' Al generally does not hold. For example, if 


1 0 i rh 
A= 0 ; and s~ |i if 
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then 


However, 


Least Squares Problems 


Definitions: 


Given A € F”*" (F = Ror C), m > n,andb € F”, the least squares problem is to find an x) € F” 
such that ||b — Ax|| is minimized: ||b — Axo|| = min ||b — Axl]. 
xeF" 


Such an Xp is called a solution to the least squares problem or a least squares solution to the linear 
system Ax = b. 

The vector r = b — Ax € F” is called the residual. 

If rank(A) = n, then the least squares problem is called the full rank least squares problem. 

If rank(A) < n, then the least squares problem is called the rank—deficient least squares problem. 


The system A* Ax = A*b is called the normal equation for the least squares problem. 
(See Chapter 39 for more information on least squares problems.) 


Facts: 
Ls 


4. 


[Mey00, p. 439] Let Ae F”*" (F = Ror C, m > n) andb € F” be given. Then the following 
statements are equivalent: 


* Xo is a solution for the least squares problem. 
* min ||b — Ax|| = ||b — Axoll. 
xeF" 
° Axo = Pb, where P is the orthogonal projection onto range(A). 
¢ A*ro = 0, where rg = b — Axy. 
° A* Ax) = A*b. 
* x) = Alb+ Yo for some yo € ker(A). 
[LH95, p. 36] If Ac F”*”" (F = RorC, m => n) and rank(A) =r < n, then x) = A'b is the 
unique solution of minimum length for the least squares problem. 
[TB97, p. 81] If Ae F™*" (F =RorC, m > n) has full rank, then x) = Atb = (A*A)~! A*b is 


the unique solution for the least squares problem. 
[TB97, p. 83] 


Algorithm 3: Solving Full Rank Least Squares via QR Factorization 
input: matrix A € F”*" (F = RorC, m > n) with full rank n and vector b € F” 
output : solution xo for minxer» ||b — Ax 

compute the reduced QR factorization A = OR; 

compute the vector c = O*b; 

solve Rxp = c using back substitution. 
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5. 


6. 


[TB97, p. 84] 


Algorithm 4: Solving Full Rank Least Squares via SVD 
input: matrix A € F”*" (F = RorC, m > n) with full rank n and vector b € F” 
output : solution x9 for minxe pn ||b — Ax! 
compute the reduced SVD A = USV* with © = diag(o},02,-++,On); 
compute the vector c = U*b; 
compute the vector y: yj = cj/oj,i = 1,2,---,n; 
compute xp = Vy. 


[TB97, p. 82] 


Algorithm 5: Solving Full Rank Least Squares via Normal Equations 
input: matrix A € F”*" (F = Ror C,m > n) with full rank n and vector b € F™ 
output : solution x9 for minxe rn ||b — Ax! 

compute the matrix A* A and the vector c = A*b; 

solve the system A* Axo = c via the Cholesky factorization. 


Examples: 


1. 


Consider the inconsistent linear system Ax = b, where 


ATA= : 


A least squares solution to the system Ax = b can be obtained via solving the normal equations: 


2/3 
5/6 


X) = (A’A)!A b= A'tb= | 


. We use Algorithm 3 to find a least squares solution of the system Ax = b given in Example 1. The 


reduced QR factorization A = QR found in Example 1 in Section 5.5 gives 


(4). 


Now solving Rx = [75,J/5]7 gives the least squares solution xp = [2/3, 5/6]". 


T 
1 


he ae 
nf 5 3/5 
AF 2. 2 
Qb=l% “38 
0 v3 3 


N 


. We use Algorithm 4 to solve the same problem given in Example 1. Using the reduced singular 


value decomposition A = UV obtained in Example 5, Section 5.6, we have 


1 5 0 


1 a 
c=U'b=——]2 6] |2 ale); 
3V5 14 _3| |3 vs 
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Now we compute y = [y1, y2]7: 


7 1 
=c,/o; = —= and = ¢)/0, = —= =. 
val 1/71 3/5 y2 2/02 2/5 


Finally, the least squares solution is obtained via 


7 
OS ee Se | mca she 5/6 
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A canonical form of a matrix is a special form with the properties that every matrix is associated to a matrix 
in that form (the canonical form of the matrix), it is unique or essentially unique (typically up to some 
type of permutation), and it has a particularly simple form (or a form well suited to a specific purpose). 
A canonical form partitions the set matrices in F”™*” into sets of matrices each having the same canonical 
form, and that canonical form matrix serves as the representative. The canonical form of a given matrix 
can provide important information about the matrix. For example, reduced row echelon form (RREF) is 
a canonical form that is useful in solving systems of linear equations; RREF partitions F’*” into sets of 
row equivalent matrices. 

The previous definition of a canonical form is far more general than the canonical forms discussed in 
this chapter. Here all matrices are square, and every matrix is similar to its canonical form. This chapter 
discusses the two most important canonical forms for square matrices over fields, the Jordan canonical 
form (and its real version) and (two versions of) the rational canonical form. These canonical forms 
capture the eigenstructure of a matrix and play important roles in many areas, for example, in matrix 
functions, Chapter 11, and in differential equations, Chapter 55. These canonical forms partition F”"*" 
into similarity classes. 

The Jordan canonical form is most often used when all eigenvalues of the matrix A € F"*” lie in the 
field F , such as when the field is algebraically closed (e.g., C), or when the field is R; otherwise the rational 
canonical form is used (e.g., for Q). The Smith normal form is a canonical form for square matrices over 
principal ideal domains (see Chapter 23); it is discussed here only as it pertains to the computation of the 
rational canonical form. If any one of these canonical forms is known, it is straightforward to determine 
the others (perhaps in the algebraic closure of the field F). Details are given in the sections on rational 
canonical form. 

Results about each type of canonical form are presented in the section on that canonical form, which 
facilitates locating a result, but obscures the connections underlying the derivations of the results. The 
facts about all of the canonical forms discussed in this section can be derived from results about modules 
over a principal ideal domain; such a module-theoretic treatment is typically presented in abstract algebra 
texts, such as [DF04, Chap. 12]. 
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None of the canonical forms discussed in this chapter is a continuous function of the entries of a matrix 
and, thus, the computation of such a canonical form is inherently unstable in finite precision arithmetic. 
(For information about perturbation theory of eigenvalues see Chapter 15; for information specifically 
about numerical computation of the Jordan canonical form, see [GV96, Chapter 7.6.5].) 


6.1 Generalized Eigenvectors 


The reader is advised to consult Section 4.3 for information about eigenvalues and eigenvectors. In this 
section and the next, F is taken to be an algebraically closed field to ensure that an n x n matrix has 
n eigenvalues, but many of the results could be rephrased for a matrix that has all its eigenvalues in F, 
without the assumption that F is algebraically closed. The real versions of the definitions and results are 
presented in Section 6.3. 


Definitions: 


Let F be an algebraically closed field (e.g., C), let A € F"*”, let z1,..., 4, be the distinct eigenvalues of 
A, and let 4 be any eigenvalue of A. 

For k a nonnegative integer, the k-eigenspace of A at A, denoted N¥(A), is ker(A — AI)*. 

The index of A at A, denoted v, (A), is the smallest integer k such that NE(A) = Nes (A). When A and 
A are clear from the context, v,(A) will be abbreviated to v, and v,,(A) to vj. 

The generalized eigenspace of A at i is the set N;(A), where v is the index of A at A. 

The vector x € F” is a generalized eigenvector of A for A ifx # 0 and x € N} (A). 

Let V be a finite dimensional vector space over F, and let T be a linear operator on V. The definitions 
of k-eigenspace of T,, index, and generalized eigenspace of T are analogous. 


Facts: 


Facts requiring proof for which no specific reference is given can be found in [HJ85, Chapter 3] or [Mey00, 
Chapter 7.8]. 

Notation: F is an algebraically closed field, A € F"*”, V is an n dimensional vector space over F, 
T € L(V,V), f1,...5 4, are the distinct eigenvalues of A or T, and A = pu; for some? € {1,...,r}. 


1. An eigenvector for eigenvalue A is a generalized eigenvector for A, but the converse is not necessarily 
true. 

2. The eigenspace for A is the 1-eigenspace, i.e., E,(A) = N, (A). 

3. Every k-eigenspace is invariant under multiplication by A. 

4. The dimension of the generalized eigenspace of A at A is the algebraic multiplicity of A, ie., 
dim N'A) = aa(u1i). 

5. A is diagonalizable if and only if v; = 1 fori = 1,...,r. 

6. F” is the vector space direct sum of the generalized eigenspaces, i.e., 


Fr= Nii (A) ® °® Nit (A). 


This is a special case of the Primary Decomposition Theorem (Fact 12 in Section 6.4). 

7. Facts 1 to 6 remain true when the matrix A is replaced by the linear operator T. 

8. If T denotes T restricted to N i (T), then the characteristic polynomial of Tis Pex) = (x- pj Jace) 
In particular, T= Lil is nilpotent. 
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Examples: 


65 18 —21 4 
—201 -—56 63 —12 
1. Let A= a = ay i € C4, pa(x) = xt + 8x? + 24x? + 32x +16 = (x +2)4, 


134 36 —42 6 


18 21 4 
gee eee ee —18] [21] [—4 
00 0 0 67| |0 0 
At+21i ,so N!,(A) = ; ; 
EES Vi ag 0 9 [80 N-2(A) = Span O67 0 
6.0) Or 0 o| lol | «7 


(A +21)? = 0, so N?,(A) = C*. Any vector not in N!,(A), e.g., e: = [1,0,0,0]’, is a generalized 
eigenvector for —2 that is not an eigenvector for —2. 


6.2 Jordan Canonical Form 


The Jordan canonical form is perhaps the single most important and widely used similarity-based canonical 
form for (square) matrices. 


Definitions: 
Let F be an algebraically closed field (e.g., C), and let A € F”*”. (The real versions of the definitions and 
results are presented in Section 6.3.) 

For A € F and positive integer k, the Jordan block of size k with eigenvalue d is the k x k matrix having 
every diagonal entry equal to A, every first superdiagonal entry equal to 1, and every other entry equal to 
0, ie., 


Xr 1 0 0 

0 2X 1 0 
IQA) = Pi 

0 0 Xx 1 

0 0 0 A 


A Jordan matrix (or a matrix in Jordan canonical form) is a block diagonal matrix having Jordan 
blocks as the diagonal blocks, ie., a matrix of the form Jz, (A1) ®--- ® Jz, (Az) for some positive integers 
t,ky,...,k, and some A,,...,A; € F. (Note: the 4; need not be distinct.) 

A Jordan canonical form of matrix A, denoted J 4 or JCF(A), is a Jordan matrix that is similar to A. It 
is conventional to group the blocks for the same eigenvalue together and to order the Jordan blocks with 
the same eigenvalue in nonincreasing size order. 

The Jordan invariants of A are the following parameters: 


* The set of distinct eigenvalues of A. 
* For each eigenvalue i, the number b, and sizes p),..., py, of the Jordan blocks with eigenvalue » 
in a Jordan canonical form of A. 


The total number of Jordan blocks in a Jordan canonical form of A is )> b,,, where the sum is taken 
over all distinct eigenvalues ju. 

If J, = C~!AC, then the ordered set of columns of C is called a Jordan basis for A. 

Let x be an eigenvector for eigenvalue A of A. If x € range(A — AI)" — range(A — 41)"*!. Then h is 
called the depth of x. 
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Let x be an eigenvector of depth h for eigenvalue A of A. A Jordan chain above x is a sequence of vectors 
Xo = X,X),...,Xp Satisfying x; = (A — AI)x;4; fori = 0,...,h—1. 

Let V be a finite dimensional vector space over F, and let T be a linear operator on V. 

A Jordan basis for T is an ordered basis B of V, with respect to which the matrix g[T]g of T is a 
Jordan matrix. In this case, g[T] is a Jordan canonical form of T, denoted JCF(T) or J, and the Jordan 
invariants of T are the Jordan invariants of JCF(T) =, [T]z. 


Facts: 


Facts requiring proof for which no specific reference is given can be found in [HJ85, Chapter 3] or [Mey00, 
Chapter 7.8]. 
Notation: F is an algebraically closed field, A, B € F"*", and A is an eigenvalue of A. 


1. A has a Jordan canonical form J 4, and J, is unique up to permutation of the Jordan blocks. In 
particular, the Jordan invariants of A are uniquely determined by A. 

2. A, B are similar if and only if they have the same Jordan invariants. 

3. The Jordan invariants and, hence, the Jordan canonical form of A can be found from the eigenvalues 
and the ranks of powers of A — AI. Specifically, the number of Jordan blocks of size k in J 4 with 
eigenvalue A is 


rank(A — AI)‘-! + rank(A — nD) — 2rank(A —AI)*. 


4. The total number of Jordan blocks in a Jordan canonical form of A is the maximal number of 
linearly independent eigenvectors of A. 

5. The number b, of Jordan blocks with eigenvalue A in J 4 equals the geometric multiplicity y4(A) 
of A. A is nonderogatory if and only if for each eigenvalue A of A, J 4 has exactly one block with A. 

6. The size of the largest Jordan block with eigenvalue 1 equals the multiplicity of 4 as a root of the 
minimal polynomial q4(x) of A. 

7. The size of the largest Jordan block with eigenvalue 4 equals the size of the index v,(A) of A at A. 

8. The sum of the sizes of all the Jordan blocks with eigenvalue A in J, (i.e., the number of times 
appears on the diagonal of the Jordan canonical form) equals the algebraic multiplicity a 4(A) of 2. 

9. Knowledge of both the characteristic and minimal polynomials suffices to determine the Jordan 
block sizes for any eigenvalue having algebraic multiplicity at most 3 and, hence, to determine the 
Jordan canonical form of A if no eigenvalue of A has algebraic multiplicity exceeding 3. This is 
not necessarily true when the algebraic multiplicity of an eigenvalue is 4 or greater (cf. Example 3 
below). 

10. Knowledge of the the algebraic multiplicity, geometric multiplicity, and index of an eigenvalue A 
suffices to determine the Jordan block sizes for 4 if the algebraic multiplicity of 4 is at most 6. This 
is not necessarily true when the algebraic multiplicity of an eigenvalue is 7 or greater (cf. Example 
4 below). 

11. The following are equivalent: 


(a) A is similar to a diagonal matrix. 
(b) The total number of Jordan blocks of A equals n. 
(c) The size of every Jordan block in a Jordan canonical form J 4 of A is 1. 


12. If A is real, then nonreal eigenvalues of A occur in conjugate pairs; furthermore, if 4 is a nonreal 
eigenvalue, then each size k Jordan block with eigenvalue 4 can be paired with a size k Jordan block 
for A. 

13. If A= A, @---@ Ay, then J4, @--- @ Ja,, is a Jordan canonical form of A. 


14. [Mey00, Chapter 7.8] A Jordan basis and Jordan canonical form of A can be constructed by using 
Algorithm 1. 
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15. 


16. 


1: 


18. 


19: 


Algorithm 1: Jordan Basis and Jordan Canonical Form 
Input: A € F”*", the distinct eigenvalues j1;,..., (4, the indices vj,..., v,. 
Output: C € F"*" such that C7! AC = Ja. 
Initially C has no columns. 
FORi = 1,...,r % working on eigenvalue j4; 
Step 1: Find a special basis B,,, for E,,; (A). 
(a) Initially B,,, has no vectors. 
(b) FOR k = v; — 1 down to 0 
Extend the set of vectors already found to a basis for range(A — pu; I ‘NE (A). 
(c) Denote the vectors of B,,, by b; (ordered as found in step (b)). 
Step 2: For each vector b; found in Step 1, build a Jordan chain above b;. 
FOR j = 1,...,dimker(A — y4;1) % working on b; 
(a) Solve (A — Mil)'uj = b; for u; where h; is the depth of bj. 
(b) Insert (A — ujI)"uj,(A — il)" !uj,...,(A — wi D)uj,u,; 
as the next h + 1 columns of C. 


A and its transpose A’ have the same Jordan canonical form (and are, therefore, similar). 

For a nilpotent matrix, the list of block sizes determines the Jordan canonical form or, equivalently, 
determines the similarity class. The number of similarity classes of nilpotent matrices of size n is 
the number of partitions of n. 

Let J 4 be a Jordan matrix, let D be the diagonal matrix having the same diagonal as J 4, and let 
N = Ja — D. Then N is nilpotent. 

A can be expressed as the sum of a diagonalizable matrix Ap and a nilpotent matrix Ay, where 
Ap and Ay are polynomials in A (and Ap and Ay commute). 

Let V be an n-dimensional vector space over F and T be a linear operator on V. Facts 1, 3 to 10, 
16, and 18 remain true when matrix A is replaced by linear operator T; in particular, JCF(T) exists 
and is independent (up to permutation of the diagonal Jordan blocks) of the ordered basis of V 
used to compute it, and the Jordan invariants of T are independent of basis. 


Examples: 


1. 


oD OW 
CoW Fe 
wre © 
i) 


J4(3) = 
O30. 0% 3 


. Let A be the matrix in Example | in Section 6.1. pa(x) = x*+8x?4+24x?+32x+16 = (x+2)4,so 


the only eigenvalue of A is —2 with algebraic multiplicity 4. From Example 1 in section 6.1, A has 3 
linearly independent eigenvectors for eigenvalue —2, so J 4 has 3 Jordan blocks with eigenvalue —2. 
—2 1 0 0 

0 -—2 0 0 

0 0 -—2 Ol" 

0 0 Q -—2 
The Jordan canonical form of A is not necessarily determined by the characteristic and minimal 
polynmials of A. For example, the Jordan matrices A = J2(0)@J1(0)@Ji(0) and B = J2(0)®J2(0) 
are not similar to each other, but have pa(x) = pg(x) = x*, qa(x) = qp(x) = x?. 
The Jordan canonical form of A is not necessarily determined by the eigenvalues and the algebraic 
multiplicity, geometric multiplicity, and index of each eigenvalue. For example, the Jordan matrices 
A = J3(0) ® J3(0) ® J, (0) and B = J3(0) ® J2(0) ® J2(0) are not similar to each other, but 
have a4(0) = ag(0) = 7,y4(0) = yp(0) = 3,v9(A) = vo(B) = 3 (and pa(x) = pa(x) = 
x’, qa(x) = qp(x) =x). 


In this case, this is enough information to completely determine that J 4 = 
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TABLE6.1  rank(A — 41)‘ 

k= 1 2 3 4 5 
A=1 11 10 9 9 9 
A=2 12 10 10 10 10 
A=3 12 11 10 9 9 


5. We use Algorithm 1 to find a matrix C such that C -lAC = J, for 


—2 3 #O 1 -l 
4 0 3 0-2 
A=| 6 —3 3 —1 —1}.Computations show that p4(x) = x° and kerA = Span(z), z2,Z3), 
—-8 6-3 2 0O 
2) 43:Y 2 1 -3 
where z; = [3, 2, —4,0,0]7,z) = [0, 1,0, —3,0]",z3 = [3,4,0,0,6]?. 
For Step 1, A? = 0, and range( A’) = Span(b;) where b; = [—1, —1,0, —1, 2)". 


Then B = {bj, z),Z} is a suitable basis (any 2 of {z1, Z2, z3} will work in this case). 
For Step 2, construct a Jordan chain above b, by solving A?u; = b,. There are many possible 


solutions; we choose u, = [0,0,0,0,1]?. Then Au, = [—1, —2, —1,0, —3]?, 
-1 -1 0 3 0 
—-1 -—2 0 2 1 0 1 (0 

C= 0 -1 0 -4 O!|, and Ja=1]0 O 1] @ [0] 6 [0]. 
-—l 0 0 0 -3 0 0 0 
—2 -3 1 0 0 


6. We compute the Jordan canonical form of a 14 x 14 matrix A by the method in Fact 3, where the 
necessary data about the eigenvalues of A and ranks is given in Table 6.1. 
A=1  — Thenumber of blocks of size 1 is 14+ 10 —2-11 = 2. 

— The number of blocks of size 2 is 11+9—2-10=0. 
— The number of blocks of size 3 is 10+ 9—2-9=1. 
So v; = 3 and b; =3. 
A =2  — The number of blocks of size 1 is 14+ 10—2-12=0. 
— The number of blocks of size 2 is 12 + 10 —2-10 = 2. 
So, vy = 2 and b2 = 2. 
24 =3  — The number of blocks of size 1 is 14+ 11—2-12= 1. 
— The number of blocks of size 2 is 12 + 10 —2-11=0. 
— The number of blocks of size 3 is 11 +9—2-10=0. 
— The number of blocks of size 4is 10+9—2-9=1. 
So, v3 = 4and b3 = 2. 


From this information, 


Ja =J3(1) ® Ji) ® Ji (1) @ J2(2) ® J2(2) ® J4(3) ® Ji (3). 


6.3 Real-Jordan Canonical Form 


The real-Jordan canonical form is used in applications to differential equations, dynamical systems, and 
control theory (see Chapter 56). The real-Jordan canonical form is discussed here only for matrices and 
with limited discussion of generalized eigenspaces; more generality is possible, and is readily derivable 
from the corresponding results for the Jordan canonical form. 
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Definitions: 


Let A € R"™", a, B,a;, 8; €R. 
The real generalized eigenspace of A at eigenvalue a + Bi is 


..__ | ker((A? — 20A+ (a? + 6)I)”) ifB #0 
EA Bi ees =ker((A—al)”) iff =0. 
The vector x € R” is a real generalized eigenvector of A for a + fi ifx #0 andx € E(A,a + Bi). 
For a, B € Rwith f ¥ 0, and even positive integer 2k, the real-Jordan block of size 2k with eigenvalue 
a £B 
a 


a+ Bi is the 2k x 2k matrix having k copies of M)(a, 8) = _B 


on the (block matrix) diagonal, k—1 


copies of Ip = I i on the first (block matrix) superdiagonal, and copies of 0) = E ; everywhere 


0 
else, i.e., 
Mp(a, B) L 05 Beg 0, 
02 M>(a, B) L ere 0, 
Tx(@ + pi) = a 
02 ae 0. M;(a, B) 0) 
02 "Re 02 02 My (aq, B) 


Areal-Jordan matrix (ora matrixin real-Jordan canonical form) is a block diagonal matrix having diag- 
onal blocks that are Jordan blocks or real-Jordan blocks, ic, a matrix of the form 
Tim, (1) +++ B Im, (ar) ® Ds (O41 + Bri) B--- @ ibs (a; + Bi) (or a permutation of the direct 
summands). 

A real-Jordan canonical form of matrix A, denoted J z or JCF8(A), is a real-Jordan matrix that is 
similar to A. It is conventional to use 6; > 0, to group the blocks for the same eigenvalue together, and to 
order the Jordan blocks with the same eigenvalue in nonincreasing size order. 

The total number of Jordan blocks in a real-Jordan canonical form of A is the number of blocks 
(Jordan or real-Jordan) in J re 


Facts: 


Facts requiring proof for which no specific reference is given can be found in [HJ85, Chapter 3]. 
Notation: A, B € R"*", a, B,0;, Bj; € R. 


1. The real generalized eigenspace of a complex number A = a + fi and its conjugate A = a — fi 
are equal, i.e., E(A,a + Bi) = E(A,a — Bi). 

2. The real-Jordan blocks with a nonreal complex number and its conjugate are similar to each other. 

3. Ahasareal-Jordan canonical form J re and J - is unique up to permutation of the diagonal (real-) 
Jordan blocks. 

4. A, B are similar if and only if their real-Jordan canonical forms have the same set of Jordan and 
real-Jordan blocks (although the order may vary). 

5. Ifall the eigenvalues of A are real, then J . is the same as J 4 (up to the order of the Jordan blocks). 

6. The real-Jordan canonical form of A can be computed from the Jordan canonical form of A. 
The nonreal eigenvalues occur in conjugate pairs, and if 8 > 0, then each size k Jordan block with 
eigenvalue a + i can be paired with a size k Jordan block for a — Bi. Then J, (a+ Bi) ® Jy (a — Bi) 
is replaced by J Fa + Bi). The Jordan blocks of J ve with real eigenvalues are the same as the those 
of J Ae 

7. The total number of Jordan and real-Jordan blocks in a real-Jordan canonical form of A is the 
number of Jordan blocks with a real eigenvalue plus half the number of Jordan blocks with a 
nonreal eigenvalue in a Jordan canonical form of A. 
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8. If B 4 0, the size of the largest real-Jordan block with eigenvalue a + fi is twice the multiplicity 
of x? — 2ax + (a? + B*) asa factor of the minimal polynomial q4(x) of A. 
9. If B 4 0, the sum of the sizes of all the real-Jordan blocks with eigenvalue w + fi in J 4 equals twice 
the algebraic multiplicity w4(a@ + £7). 
10. If 8 £0, dim E(A,a + Bi) = aa(a + Bi). 
ll. (A= A; ®--:@ Am, then Te @:-:-@® ios is a real-Jordan canonical form of A. 


Examples: 
-10 6 -4 4 0 
-17 10 -4 6 -1 
l. Leta=| -4 2 -—3 2 1] . Since the characteristic and minimal polynomials of A are 
—-ll 6 -l1 6 3 
-4 2 -4 2 2 
both x° — 5x4 + 12x? — 16x? + 12x — 4 = (x — 1) (x? -— 2x +2)’, 
1 1 1 0 0 
—-1 1 0 1 0 
i ge a a 
0 0 -1 1 0 
0 0 0 0 1 0010 
2. The requirement that 6 # 0 is important. A = : ; : : is not a real-Jordan matrix; 
0 0 0 0 


Ja 


II 
cooo°o 
oooFr 
ooo°o 
oroeo 


6.4 Rational Canonical Form: Elementary Divisors 


The elementary divisors rational canonical form is closely related to the Jordan canonical form (see Fact 
7 below). A rational canonical form (either the elementary divisors or the invariant factors version, cf. 
Section 6.6) is used when it is desirable to stay within a field that is not algebraically closed, such as the 
rational numbers. 


Definitions: 


Let F bea field. 
For a monic polynomial p(x) = x" + ¢y—1x"7! Fee + Cox? + 1x + co € F[x] (with n > 1), the 


0 0 bales —Co 

1 0 yas —C] 
companion matrix of p(x) is then x n matrix C(p) = 

QO «-. 1 —Cha1 


An elementary divisors rational canonical form matrix (ED-RCF matrix) (over F ) is a block diagonal 
matrix of the form C(h{") ® --» ® C(h;") where each h;(x) is a monic polynomial that is irreducible 
over F. 
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The elementary divisors of the ED-RCF matrix C(h}") ®--- ® C(h;") are the polynomials h;(x)™, 
H3 ooct 

An elementary divisors rational canonical form of matrix A € F”*", denoted RCFgp(A), is an ED- 
RCF matrix that is similar to A. It is conventional to group the companion matrices associated with powers 
of the same irreducible polynomial together, and within such a group to order the blocks in size order. 

The elementary divisors of A are the elementary divisors of RCFgp(A). 

Let V be a finite dimensional vector space over F, and let T be a linear operator on V. 

An ED-RCF basis for T is an ordered basis B of V, with respect to which the matrix g[T]g of T is 
an ED-RCF matrix. In this case, g[T]g is an elementary divisors rational canonical form of T, denoted 
RCFgp(T), and the elementary divisors of T are the elementary divisors of RCFgp(T) =g [T]p. 

Let q(x) be a monic polynomial over F. 

A primary decomposition of a nonconstant monic polynomial q(x) over F is a factorization q(x) = 
(hy (x))" +--+ (h,(x)), where the h;(x),i = 1,...,r are distinct monic irreducible polynomials over F. 

The factors (h;(x))” in a primary decomposition of q(x) are the primary factors of q(x). 


Facts: 


Facts requiring proof for which no specific reference is given can be found in [HK71, Chapter 7] or [DF04, 
Chapter 12]. 


1. The characteristic and minimal polynomials of the companion matrix C(p) are both equal to p(x). 

2. Whether or not a matrix is an ED-RCF matrix depends on polynomial irreducibility, which depends 
on the field. See Example 1 below. 

3. Every matrix A € F”*”" is similar to an ED-RCF matrix, RCFgp(A), and RCFgp(A) is unique 
up to permutation of the companion matrix blocks on the diagonal. In particular, the elementary 
divisors of A are uniquely determined by A. 

4. A,B € F"*" are similar if and only if they have the same elementary divisors. 

5. (See Fact 3 in Section 6.2) For A € F"*", the elementary divisors and, hence, RCFgp(A) can be 
found from the irreducible factors h;(x) of the characteristic polynomial of A and the ranks of 
powers of h;(A). Specifically, the number of times h; (x)k appears as an elementary divisor of A is 


— + _ (ank(ni( Ay)! + rank(h;( A) — 2rank(hi(A))*). 
deg hi (x) 
6. If A € F"*” has n eigenvalues in F, then the elementary divisors of A are the polynomials (x — A)‘, 
where the J;(A) are the Jordan blocks of J 4. 
7. There is a natural association between the diagonal blocks in the elementary divisors rational 
canonical form of A and the Jordan canonical form of A in F"*", where F is the algebraic closure 
of F. Let h(x)” be an elementary divisor of A, and factor h(x) into monic linear factors over F, 
h(x) = (x — A,)--- (x — A). If the roots of h(x) are distinct (e.g., if the characteristic of F is 0 
or F isa finite field), then the ED-RCF diagonal block C(h™) is associated with the Jordan blocks 
Jm(Aj),i = 1,...,t. If the characteristic of F is p and h(x) has repeated roots, then all roots have 
the same multiplicity p* (for some positive integer k) and the ED-RCF diagonal block C(h’) is 
associated with the Jordan blocks J ptmQi)st a Pera 
8. [HK71, Chapter 4.5] Every monic polynomial q(x) over F has a primary decomposition. The 
primary decomposition is unique up to the order of the monic irreducible polynomials, i-e., the 
set of primary factors of q(x) is unique. 
9. [HK71, Chapter 6.8] Let q(x) € F[x], let hj(x)”,i = 1,...,1r be the primary factors, and define 
filx) = ae . Then there exist polynomials g;(x) such that fi(x)gi(x) +---+ f-(x)g-(x) = 1. 
Let A € F"*" and let qa(x) = (hy(x))™---(h,(x))" be a primary decomposition of its minimal 
polynomial. 
10. Every primary factor h;(x)" of q4(x) is an elementary divisor of A. 
11. Every elementary divisor of A is of the form (h;(x))” with m < m; for somei € {1,...,r}. 
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12. [HK71, Chapter 6.8] Primary Decomposition Theorem 

(a) F" =ker(h,(A)"™) ®--- @ ker(h, (A)””). 

(b) Let f; and g; be as defined in Fact 9. Then fori = 1,...,r, Ej = f;(A)gj(A) is the projection 
onto ker(h;(A)’”) along ker(hy(A)™) @ --- @ ker(hj_)(A)"*"') ® ker(hj41(A)"*') @--- ® 
ker(h,(A)”"). 

(c) The E; = f;(A)gi(A) are mutually orthogonal idempotents (i.e., E? = E; and E;E; = Oif 
i#jjandI = E,+---+E,. 


13. [HK71, Chapter 6.7] If A € F”*" is diagonalizable, then A = ww, E, +---+ ER where the 

E; are the projections defined in Fact 12 with primary factors hj(x)"" = (x — pj) of qa(x). 
Let V be an n-dimensional vector space over F, and let T be a linear operator on V. 

14. Facts 3,5 to 7, and 10 to 13 remain true when matrix A is replaced by linear operator T; in particular, 
RCFgp(T) exists and is independent (up to permutation of the companion matrix diagonal blocks) 
of the ordered basis of V used to compute it, and the elementary divisors of T are independent of 
basis. 

15. If T denotes T restricted to ker(h;(T)” ), then the minimal polynomial of T is hj(T)™. 


Examples: 
comes: 0 2 : ; ; se 
1. LetA=[-1]@[-1]@6]1 0 -3/1 6 i: @ Oe Ge 4 . Over Q, A is an ED-RCF 
ak a 001 0 


matrix and its elementary divisors are x + 1,x +1, (x +1)°, x? — 2, (x? — 2). Ais not an ED-RCF 
matrix over C because x” — 2 is not irreducible over C. 


—l 1 0 
_ J2 1 -/2 1 
JCF(A)=[—1] @ [-1] ® : ss : @lV27@l-V6]y Bl®] 9 _ ap 
where the order of the Jordan blocks has been chosen to emphasize the connection to 


RCFgp(A) = A. 


2. Let A= 0 0 O 0 € Q°*°. We use Fact 5 to determine the elementary divisors 


wo BRR OF 


0 0 -l 0 
rational canonical form of A. The following computations can be performed easily over Q in a 
computer algebra system such as Mathematica, Maple, or MATLAB® (see Chapters 71, 72, 73), or 
on a matrix-capable calculator. p4(x) = x° — 3x4 + x? + 5x? — 6x +2 = (x — 1)? (x? 2). 
Table 6.2 gives the of ranks h; (A)* where h;(x) is shown in the left column. 
h(x) =x —1 The number of times x — 1 appears as an elementary divisor is 5+ 2—2-3= 1. 


The number of (x — 1)? appears as an elementary divisor is 3 + 2—2-2=1. 
h(x) = x? — 2 The number of times x” — 2 appears as an elementary divisor is (5+ 3—2-3)/2 = 1. 


Thus, RCFgp(A) = C(x — 1) ® C(x — 1)?) ® C(x? — 2) = [1] ® : qe : | 


TABLE 6.2 rank(h(A)*) 

c= 1 2 3 
hi(x)=x-1 3 2 2 
ho(x)=x?-2 3 3 3 


Canonical Forms 6-11 


3. We find the projections E;,i = 1,2 in Fact 12 for A in the previous example. From the ele- 
mentary divisors of A, qa(x) = (x — 1)?(x? — 2). Let hy(x) = (x — 1)*,Ao(x) = x* — 2. Then 
filx) = x?—2, fo(x) = (x—1)?. Note: normally the f;(x) will not be primary factors; this happens 


here because there are only two primary factors. If we choose gi(x) = —(2x — 1), go(x) = 2x +3, 
then 1 = fi(x)gi(x) + fo(x)go(x) (g1, g2 can be found by the Euclidean algorithm). Then 

—2 1 -1 1 #0 3 -1 1 -1 0 

0 0 0 0 0 0 1 O 0 0 

E,=fi(Agi(A)=] 0 0 1 0 0 and E, = f,(A)go(A) = |0 0 0 0 0 

—-6 3 —2 3 0 6 -—3 2 -—2 0 

0 0 0 0 1 0 0 O 0 0 


and it is easy to verify that E? = Ey, E} = Ey, E,E, = E,E,; =0,and E;+ £,=1. 


6.5 Smith Normal Form on F [x]"*” 


For a matrix A € F”*", the Smith normal form of xJ,, — A is an important tool for the computation of 
the invariant factors rational canonical form of A discussed in Section 6.6. In this section, Smith normal 
form is discussed only for matrices in F [x]"*", and the emphasis is on finding the Smith normal form of 
xI, — A, where A € F"*". Smith normal form is used more generally for matrices over principal ideal 
domains (see Section 23.2); it is not used extensively as a canonical form within F”*", since the Smith 
normal form of a matrix A € F”"*" of rank k is Ik ® On. 


Definitions: 


Let F be a field. For M € F[x]"*", the following operations are the elementary row and column 
operations on M: 


(a) Interchange rows i, j, denoted Rj <> Rj (analogous column operation denoted C; < Cj). 

(b) Adda p(x) multiple of row j to rowi, denoted R; + p(x)R; > R; (analogous column operation 
denoted C; + p(x)Cj; > Cj). 

(c) Multiply row i by a nonzero element b of F, denoted bR; — R; (analogous column operation 
denoted bC; > C;). 


A Smith normal matrix in F[x]"*" is a diagonal matrix D = diag(1,...,1,a)(x),...,a;(x),0, 
...,0), where the a;(x) are monic nonconstant polynomials such that aj(x) divides aj4)(x) fori = 
1,...,5—1. 

The Smith normal form of M € F [x]"*" is the Smith normal matrix obtained from M by elementary 
row and column operations. 

For A € F”*", the monic nonconstant polynomials of the Smith normal form of x1, — A are the Smith 
invariant factors of A. 


Facts: 


Facts requiring proof for which no specific reference is given can be found in [HK71, Chapter 7] or [DF04, 
Chapter 12]. 


1. Let M € F[x]"*". Then M has a unique Smith normal form. 

2. Let A € F"*". There are no zeros on the diagonal of the Smith normal form of xI, — A. 

3. (Division Property) If a(x), b(x) € F[x] and b(x) 4 0, then there exist polynomials q(x), r(x) 
such that a(x) = q(x)b(x) + r(x) and r(x) = 0 or degr(x) < deg b(x). 

4. The Smith normal form of M = xI — Aand, thus, the Smith invariant factors of A can be computed 
as follows: 
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* Fork =1,...,n—1 


— Use elementary row and column operations and the division property of F [x] to place 
the greatest common divisor of the entries of M[{k, ...,}] in the kth diagonal position. 


— Use elementary row and column operations to create zeros in all nondiagonal positions 
in row k and column k. 


¢ Make the nth diagonal entry monic by multiplying the last column by a nonzero element of 
Es 


This process is illustrated in Example 1 below. 


Examples: 


1. Lett A= . We use the method in Fact 4 above to find the Smith normal form of 


LN Oe 
CoW Fr 
i Oman! 

| 


0 6 —3 

M =xI-— Aand invariant factors of A. 

¢ k = 1: Use the row and column operations on M (in the order shown): 
Ry @ R3,-3R, > Ry, R3+ (1 —x)Ri > Rs, Ra + 4Ri > Ra, 
C3 +(-2+ 5)Cy > C3,C4+C, > Cy 


1 0 0 0 
: 0 —3 —2 2 
to obtain M, = . Bh 
0 -1 2-41 x 
0 0 2 — 2x x-1 


° k = 2: Use the row and column operations on M, (in the order shown): 
R3 <> R, —-1R) —? R, R3 + (3 — x)R> rea R3, 
C3+(1- sey ENG —> C3,C4+xCy > C4 


L 0 0 0 

ere 0 1 0 0 

to obtain M, = ° 0 2 ay 178 5 g 3x 42 . 
0 0 2—2x x-1 


° k = 3 (and final step): Use the row and column operations on M, (in the order shown): 
R3 <> Ra, —}R3 > R3, Ra + 54 (x — 2)(x — 5)R3 > Ra, 
Cy + $C3 > C4, 4Cq > Cy 


1 0 0 0 
. . 0 1 0 0 
to obtain the Smith normal form of M, M3 = 
0 0x-I1 0 
00 OO x3—4x?4+5x—2 


The Smith invariant factors of A are x — 1, x° — 4x* + 5x — 2. 


6.6 Rational Canonical Form: Invariant Factors 


Like the elementary divisors version, the invariant factors rational canonical form does not require the field 
to be algebraically closed. It has two other advantages: This canonical form is unique (not just unique up 
to permutation), and (unlike elementary divisors rational canonical form) whether a matrix is in invariant 
factors rational canonical form is independent of the field (see Fact 2 below). 
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Definitions: 


Let F bea field. An invariant factors rational canonical form matrix (IF-RCF matrix) is a block diagonal 
matrix of the form C(a,) @ --- 6 C(a,), where a;(x) divides a;,,(x) fori = 1,...,s —1. 

The invariant factors of the IF-RCF matrix C(a,) @--- @ C(a,) are the polynomials a;(x),i = 1,...s. 

The invariant factors rational canonical form of matrix A € F”*", denoted RCF;p (A), is the IF-RCF 
matrix that is similar to A. 

The invariant factors of A are the invariant factors of RCF;; (A). 

Let V be a finite dimensional vector space over F and let T be a linear operator on V. 

An IF-RCF basis for T is an ordered basis B of V, with respect to which the matrix g[T]g of T is 
an IF-RCF matrix. In this case, 3[T], is the invariant factors rational canonical form of T, denoted 
RCF,;f(T), and the invariant factors of T are the invariant factors of RCF;7(T) =g [T]g. 


Facts: 


Facts requiring proof for which no specific reference is given can be found in [HK71, Chapter 7] or [DF04, 
Chapter 12]. Notation: A € F"™*", 


1. Every square matrix A is similar to a unique IF-RCF matrix, RCF;f (A). 

2. RCFyp(A) is independent of field. That is, if K is an extension field of F and A is considered as an 

element of K”*", RCF;; (A) is the same as when A is considered as an element of F”*”. 

Let B € F"*". Then A, B are similar if and only if RCF;r(A) = RCF;r(B). 

The characteristic polynomial is the product of the invariant factors of A, i.e., p4(x) = a(x) ---a,(x). 

The minimal polynomial of A is the invariant factor of highest degree, ie., a(x) = a; (x). 

The elementary divisors of A € F"*” are the primary factors (over F) of the invariant factors 

of A. 

7. The Smith invariant factors of A are the invariant factors of A. 

8. [DFO04, Chapter 12.2] RCF;7 (A) anda nonsingular matrix S € F"*" such that S~'AS = RCF;r (A) 
can be computed by Algorithm 2. 


De 


Algorithm 2: Rational Canonical Form (invariant factors) 

1. Compute the Smith normal form D of M = xI — Aas in Fact 4 of section 6.5, 
keeping track of the elementary row operations, in the order performed (column 
operations need not be recorded). 


2. The invariant factors are the nonconstant diagonal elements a;(x),...,a;(x) of D. 
3. Let dj,...,d, denote the degrees of a;(x),..., a5 (x). 

4.LetG =I. 

5. FORk = 1,..., number of row operations performed in step 1 


(a) If the kth row operation is R; < Rj, then perform column operation C; = C; on G. 
(b) If the kth row operation is R; + p(x)R; — R;, then perform column operation 
Cj — p(A)C; > C; on G (note index reversal). 
(c) If the kth row operation is bR; — R;, then perform column operation 
-Ci —> CjonG. 
6. G will have 0s in the first n — s columns; denote the remaining columns of G by gi,...,g;. 
. Initially S has no columns. 
8. FORk = 1,...,5 
(a) Insert g, as the next column of S (working left to right). 
(b) FORi = 1,...,aq—1. 
Insert A times the last column inserted as the next column of S. 
9. RCF;-(A) = S7!AS. 


N 
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9. Let V be an n-dimensional vector space over F, and let T bea linear operator on V. Facts 1, 2, 4 to 
6 remain true when matrix A is replaced by linear operator T; in particular, RCF;(T) exists and 
is unique (independent of the ordered basis of V used to compute it). 


Examples: 


1. We can use the elementary divisors already computed to find the invariant factors and IF-RCF of A 
in Example 2 of Section 6.4. The elementary divisors of A are x — 1, (x — 1)*, x? — 2. We combine 
these, working down from the highest power of each irreducible polynomial. 
ay(x) = (x — 1)? (x? — 2) = x* — 2x3 — x? + 4x — 2,a;(x) = x — 1. Then 


1000 0 
0000 2 
RCE;p(A) = C(x — 1) @ C(x* — 2x7 —x?+4x—-2)=]0 1 0 0 —-4 
0010 1 
0001 2 


2. By Fact 7, for the matrix A in Example 1 in Section 6.5, RCF;f(A) = C(x—1)®C(x?—4x?+5x—2). 
3. We can use Algorithm 2 to find a matrix S$ such that RCF;r(A) = S —1AS for the matrix A in 
Example 1. 


¢ k = 1: Starting with G = I4, perform the column operations (in the order shown): 


Cy ad C3, —2C\ > Ci, Cy (14 A)C3 > Ci, Cy 4C4 > Ci, 


0 0 1 0 
. 0 1 0 0 

to obtain G, = Oe B00 2ole 
0 0 0 1 


¢ k = 2: Use column operations on G (in the order shown): 


C3 > C2, —-1C, = C), Cy = (314 = A)C3 2: C2, 


0 0 0 0 
, 0 0 1 0 

to obtain G = GenetGe oe 
0 00 1 


¢ k = 3 (and final step of Fact 4 in Section 6.5): 
Use column operations on G (in the order shown): 


C3 <> C4, —2C3 > C3, C3 + 5(A — 214)(A — 514)Cy > Cs, 


00 -3 0 
0 -1 1 
to obtain G = [g), 89, 83,84] = 0 0 1 0 
0 0 0 0 
-2 014 10 0 4 
- 2 = — o-l — 
Then S = [g3, gs, Aga, A°ga] = 1002 and RCF7p(A) = SAS = 0 10 —5]° 
000 4 001 “| 
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Unitary transformations preserve the inner product. Hence, they preserve the metric quantities that stem 
from the inner product, suchas length, distance, and angle. While a general similarity preserves the algebraic 
features of a linear transformation, such as the characteristic and minimal polynomials, the rank, and the 
Jordan canonical form, unitary similarities also preserve metric features such as the norm, singular values, 
and the numerical range. Unitary similarities are desirable in computational linear algebra for stability 
reasons. 

Normal transformations are those which have an orthogonal basis of eigenvectors and, thus, can be 
represented by diagonal matrices relative to an orthonormal basis. The class of normal transformations 
includes Hermitian, skew-Hermitian, and unitary transformations; studying normal matrices leads to a 
more unified understanding of all of these special types of transformations. Often, results that are dis- 
covered first for Hermitian matrices can be generalized to the class of normal matrices. Since normal 
matrices are unitarily similar to diagonal matrices, things that are obviously true for diagonal matrices 
often hold for normal matrices as well; for example, the singular values of a normal matrix are the absolute 
values of the eigenvalues. Normal matrices have two important properties — diagonalizability and an 
orthonormal basis of eigenvectors — that tend to make life easier in both theoretical and computational 
situations. 


7.1 Unitary Similarity 


In this subsection, all matrices are over the complex numbers and are square. All vector spaces are finite 
dimensional complex inner product spaces. 


Definitions: 


A matrix U is unitary if U*U = I. 
A matrix Q is orthogonal if Q'Q=I. 
Note: This extends the definition of orthogonal matrix given earlier in Section 5.2 for real matrices. 


7-1 
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Matrices A and B are unitarily similar if B = U* AU for some unitary matrix U. The term unitarily 
equivalent is sometimes used in the literature. 

The numerical range of A is W(A) = {v* Av|v*v = 1}. 

The Frobenius (Eulidean) norm of the matrix A is || Al| - = (iia \aij eee = (tr(A* A)". (See 
Chapter 37 for more information on norms.) 

The operator norm of the matrix A induced by the vector 2-norm ||-||2 is || Allz = max{|| Av||||||vll = 1} 
this norm is also called the spectral norm. 


Facts: 


Most of the material in this section can be found in one or more of the following: [HJ85, Chap. 2] 
[Hal87, Chap. 3] [Gan59, Chap. IX] [MM64, 1.4, II.5]. Specific references are also given for some 
facts. 


1. A real, orthogonal matrix is unitary. 
2. The following are equivalent: 


* U is unitary. 

* U is invertible and U~! = U*. 

* The columns of U are orthonormal. 

* The rows of U are orthonormal. 

* For any vectors x and y, we have (Ux, Uy) = (x,y). 


* For any vector x, we have || Ux|| = ||x||. 


3. If U is unitary, then U*, U', and U are also unitary. 

4. If U is unitary, then every eigenvalue of U has modulus 1 and | det(U)| = 1. Also, ||U||2 = 1. 

5. The product of two unitary matrices is unitary and the product of two orthogonal matrices is 
orthogonal. 

6. The set of n x n unitary matrices, denoted U(n), is a subgroup of GL(n,C), called the unitary 
group. The subgroup of elements of U(n) with determinant one is the special unitary group, 
denoted SU(n). Similarly, the set of n x n real orthogonal matrices, denoted O(n), is a subgroup 
of GL(n,R), called the real, orthogonal group, and the subgroup of real, orthogonal matrices of 
determinant one is SO(n), the special orthogonal group. 

7. Let U be unitary. Then 


° Alle = |U*AU |e. 
* All, = ]U*AU]}2. 
* Aand U*AU have the same singular values, as well as the same eigenvalues. 


° W(A) = W(U* AU). 


8. [Sch09] Any square, complex matrix A is unitarily similar to a triangular matrix. If T = U*AU 
is triangular, then the diagonal entries of T are the eigenvalues of A. The unitary matrix U 
can be chosen to get the eigenvalues in any desired order along the diagonal of T. Algorithm 1 
below gives a method for finding U, assuming that one knows how to find an eigenvalue and 
eigenvector, e.g., by exact methods for small matrices (Section 4.3), and how to find an or- 
thonormal basis containing the given vector, e.g., by the Gram-Schmidt process (Section 5.5). 
This algorithm is designed to illuminate the result, not for computation with large matrices in 
finite precision arithmetic; for such problems appropriate numerical methods should be used 
(cf. Section 43.2). 
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Algorithm 1: Unitary Triangularization 
Input: A € C"*”, 
Output: unitary U such that U* AU = T is triangular. 
1A, =A. 
2.FORkK=1,...,5n—1 
(a) Find an eigenvalue and normalized eigenvector x of the (n + 1 —k) x (n+1—k) 
matrix Ax. 
(b) Find an orthonormal basis x, yo,... 5 ¥n41—« for Crti-k 
(c) Ux = [% ya. +--+ Yn41—kl- 
(d) Uk = Ik-1 ® Ux (U; = Uj). 
(e) By = Uf Ay Ug. 
(f) Axyi = B;,(1), the (n — k) x (n — k) matrix obtained from B, by deleting the first row 
and column. 


2 U= UU55 vie Ua han 


9. (A strictly real version of the Schur unitary triangularization theorem) If A is a real matrix, then 
there is a real, orthogonal matrix Q such that Q' AQ is block triangular, with the blocks of size 
1 x lor 2 x 2. Each real eigenvalue of A appears as a1 x 1 block of Q? AQ and each nonreal pair 
of complex conjugate eigenvalues corresponds to a 2 x 2 diagonal block of Q7 AQ. 

10. If F is a commuting family of matrices, then F is simultaneously unitarily triangularizable — i.e., 
there is a unitary matrix U such that U* AU is triangular for every matrix A in F. This fact has the 
analogous real form also. 

11. [Lit53] [Mit53] [Sha91] Let 41,42,---,A, be the distinct eigenvalues of A with multiplicities 
M,,M,--+ ,m,. Suppose U* AU is block triangular with diagonal blocks Aj, Ao,..., Ay, where A; 
is size m; x mj; and A; is the only eigenvalue of A; for each i. Then the Jordan canonical form of A 
is the direct sum of the Jordan canonical forms of the blocks Aj, A2,..., A;. Note: This conclusion 
also holds if the unitary similarity U is replaced by an ordinary similarity. 

12. Let Aj,A2,--- , A, be the eigenvalues of the n x n matrix A and let T = U* AU be triangular. Then 
Alle = Di AVP? + 0; 2; [aj /?. Hence, || Alle = S07, |Ai/? and equality holds if and only if T 
is diagonal, or equivalently, if and only if A is normal (see Section 7.2). 


13. A2 x 2 matrix A with eigenvalues 4, A, is unitarily similar to the triangular matrix E i | : 
2 


where r = VIlAlk — (Ai |? + |A2|*). Note that r is real and nonnegative. 
14. Two 2 x 2 matrices, A and B, are unitarily similar if and only if they have the same eigenvalues and 


|Allz = |Blle. 
15. Any square matrix A is unitarily similar to a matrix in which all of the diagonal entries are equal 
tr (A) 
to 2 


16. [Spe40] Two n x n matrices, A and B, are unitarily equivalent if and only if tr w(A, A*) = 
tr w(B, B*) for every word w(s, t) in two noncommuting variables. 

17. [Pea62] Two n x n matrices, A and B, are unitarily equivalent if and only if tr w(A, A*) = 
tr w(B, B*) for every word w(s, t) in two noncommuting variables of degree at most 2n?. 


Examples: 


. 1 fl ; : 
1. The matrix — f is unitary but not orthogonal. 
i 


J2 
1 
V1 + 2i 


1 14+1 


2. The matrix ike 4 | is orthogonal but not unitary. 
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4 1 
3. Fact 13 shows that A = 0 i , 


1}. Batt held 
] is unitarily similar to A = 


4. For any nonzero r, the matrices F 4 and F ) are similar, but not unitarily similar. 
—31 21 48 
5. LetA=|]-—4 4. 6].Apply Algorithm 1 to A: 
—20 13 31 
Step 1. A; = A. 
Step 2. For 


k=1:(a) pa,(x) = x? — 4x? +5x —2 = (x — 2)(x — 1)’, so the eigenvalues are 1, 1, 
2. From the reduced row echelon form of A — I3, we see that [3,0,2]" is an 
eigenvector for 1 and, thus, x = [Fs 0, gal" is a normalized eigenvector. 

(b) One expects to apply the Gram—Schmidt process to a basis that includes x as 
the first vector to produce an orthonormal basis. In this example, it is obvious 


how to find an orthonormal basis for C?: 

23h {Qh 

VB VB 
() U=;0 1 0 
2 
VB 


(d) unnecessary. 


89 
1 scl 68 
(e B, — UF AU, — 0 4 2 13 + 
3 


4 2/713 
(f) A, = _ 3 4 } 
k =2:(a) 1is still an eigenvalue of A2. From the reduced row echelon form of A, — h, 


we see that [—2./13, 3]” is an eigenvector for 1 and, thus, x = [—2,/ 2, al” 
is a normalized eigenvector. 


(b) Again, the orthonormal basis is obvious: 


13 3 
_|--Va ai 
3 


(c) U; 
B 
Vel 61 
1 0 0 
2 1B 3 
d@ith=|Sr "Va ya 
3 B 
0 ya Wa 
} —22 
(ec) B= ale 
0 2 
(f) unnecessary. 
3 7 - 1 26, 2035, 
13 793 61 al a3 
—tTtth— 1B 3 = = 29° 
Step 3. U=U,U,= | 0 2/5 5 .T =U*AU 0 1 =|: 
2 9 6 0 0 2 


13 793 61 
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Fe 


[HJ85, p. 84] Schur’s theorem tells us that every complex, square matrix is unitarily similar to 
a triangular matrix. However, it is not true that every complex, square matrix is similar to a 
triangular matrix via a complex, orthogonal similarity. For, suppose A = QT QV", where Q is 
complex orthogonal and T is triangular. Let q be the first column of Q. Then q is an eigenvector of 
i 


: 1 ; Ie ss 
A and q'q = 1. However, the matrix A = |. , has no such eigenvector; A is nilpotent and 
i 


F : : 1 
any eigenvector of A is a scalar multiple of | 5 


Normal Matrices and Spectral Theory 


In this subsection, all matrices are over the complex numbers and are square. All vector spaces are finite 


dimensional complex inner product spaces. 


Definitions: 


The matrix A is normal if AA* = A* A. 

The matrix A is Hermitian if A* = A. 

The matrix A is skew-Hermitian if A* = — A. 

The linear operator, T’, on the complex inner product space V is normal if TT* = T*T. 

Two orthogonal projections, P and Q, are pairwise orthogonal if PQ = QP = 0. (See Section 5.4 for 
information about orthogonal projection.) 

The matrices A and B are said to have Property L if their eigenvalues ax, Bx, (k = 1,---,n) may be 
ordered in such a way that the eigenvalues of x A + yB are given by xa; + yx for all complex numbers x 


and y. 


Facts: 


Most of the material in this section can be found in one or more of the following: [HJ85, Chap. 2] [Hal87, 
Chap. 3] [Gan59, Chap. IX] [MM64, I.4, III.3.5, III.5] [GJSW87]. Specific references are also given for 
some facts. 


Is 


Diagonal, Hermitian, skew-Hermitian, and unitary matrices are all normal. Note that real symmet- 
ric matrices are Hermitian, real skew-symmetric matrices are skew-Hermitian, and real, orthogonal 
matrices are unitary, so all of these matrices are normal. 


. If U is unitary, then A is normal if and only if U* AU is normal. 
. Let T bea linear operator on the complex inner product space V. Let B be an ordered orthonormal 


basis of V and let A = [T],. Then T is normal if and only if A is a normal matrix. 


. (Spectral Theorem) The following three versions are equivalent. 


¢ A matrix is normal ifand only ifit is unitarily similar to a diagonal matrix. (Note: This is sometimes 
taken as the definition of normal. See Fact 6 below for a strictly real version.) 

* The matrix A is normal if and only if there is an orthonormal basis of eigenvectors of A. 

© Let Aj,A2,... A, be the distinct eigenvalues of A with algebraic multiplicities m7, m2,... ,m. 
Then A is normal if and only if there exist t pairwise orthogonal, orthogonal projections 
P, Py,..., P; such that }7}_, P; = I, rank(P;) = m;, and A = S7j_, A; P;. (Note that the two 
orthogonal projections P and Q are pairwise orthogonal if and only if range(P) and range(Q) 
are orthogonal subspaces.) 


. (Principal Axes Theorem) A real matrix A is symmetric if and only if A = QDQ’, where Q isa 
p Y: 'Y 


real, orthogonal matrix and D is a real, diagonal matrix. Equivalently, a real matrix A is symmetric 
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if and only if there is a real, orthonormal basis of eigenvectors of A. Note that the eigenvalues of 
A appear on the diagonal of D, and the columns of Q are eigenvectors of A. The Principal Axes 
Theorem follows from the Spectral Theorem, and the fact that all of the eigenvalues of a Hermitian 
matrix are real. 


. (A strictly real version of the Spectral Theorem) If A is a real, normal matrix, then there is a real, 


orthogonal matrix Q such that Q!™ AQ is block diagonal, with the blocks of size 1 x 1 or 2 x 2. 
Each real eigenvalue of A appears as a 1 x 1 block of Q? AQ and each nonreal pair of complex 
conjugate eigenvalues corresponds to a 2 x 2 diagonal block of Q7 AQ. 


. The following are equivalent. See also Facts 4 and 8. See [GJSW87] and [EI98] for more equivalent 


conditions. 


¢ Ais normal. 

¢ A* can be expressed as a polynomial in A. 

* For any B, AB = BA implies A*B = BA*. 

* Any eigenvector of A is also an eigenvector of A”. 

* Each invariant subspace of A is also an invariant subspace of A*. 

* Foreach invariant subspace, V, of A, the orthogonal complement, V+, isalso an invariant subspace 
of A. 

° (Ax, Ay) = (A*x, A*y) for all vectors x and y. 

° (Ax, Ax) = (A*x, A*x) for every vector x. 

* || Ax|| = || A*x|| for every vector x. 

¢ A* = UA for some unitary matrix U. 

° || All: = Ii \?, where A}, A2,--+ 5A, are the eigenvalues of A. 

* The singular values of A are |Aj|, |A2|,--- 5 |An|, where A1,A2,--- , An are the eigenvalues of A. 

° If A= UP isa polar decomposition of A, then UP = PU. (See Section 8.4.) 

* A commutes with a normal matrix with distinct eigenvalues. 

¢ A commutes with a Hermitian matrix with distinct eigenvalues. 

¢ The Hermitian matrix AA*— A* A is semidefinite (i.e., it does not have both positive and negative 


eigenvalues). 


A* A— A* 


A 
. LetH = = and K = a Then H and K are Hermitian and A = H +iK. The matrix 
i 


A is normal if and only if HK = KH. 


. If Ais normal, then 


* Ais Hermitian if and only if all of the eigenvalues of A are real. 
* Ais skew-Hermitian if and only if all of the eigenvalues of A are pure imaginary. 


* Ais unitary if and only if all of the eigenvalues of A have modulus 1. 


The matrix U is unitary if and only if U = exp(iH) where H is Hermitian. 

If Q is a real matrix with det(Q) = 1, then Q is orthogonal if and only if Q = exp(K), where K is 
a real, skew-symmetric matrix. 

(Cayley’s Formulas/Cayley Transform) If U is unitary and does not have —1 as an eigenvalue, then 
U = (1+iH)(I —iH)~!, where H = i(I — U)(I + U)7' is Hermitian. 

(Cayley’s Formulas/Cayley Transform, real version) If Q is a real, orthogonal matrix which does 
not have —1 as an eigenvalue, then Q = (I — K)(I + K)~', where K = (I — Q)(I+ Q)"'isa 
real, skew-symmetric matrix. 

A triangular matrix is normal if and only if it is diagonal. More generally, if the block triangular 


By By 


matrix, | 0 8B | (where the diagonal blocks, B;;,i = 1,2, are square), is normal, then By = 0. 
22 
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21. 
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Let A be a normal matrix. Then the diagonal entries of A are the eigenvalues of A if and only if A 
is diagonal. 

If A and B are normal and commute, then AB is normal. However, the product of two noncom- 
muting normal matrices need not be normal. (See Example 3 below.) 

If A is normal, then (A) = || Al|2. Consequently, if A is normal, then (A) = |a;;| for alli and j. 
The converses of both of these facts are false (see Example 4 below). 

[MM64, p. 168] [MM55] [ST80] If A is normal, then W(A) is the convex hull of the eigenvalues 
of A. The converse of this statement holds when n < 4, but not for n > 5. 


[WW49] [MM64, page 162] Let A be anormal matrix and suppose x is a vector such that (Ax); = 0 


Ax); . 
whenever x; = 0. For each nonzero component, x;, of x, define uj = aay . Note that 1; is a 
aes 


j 
complex number, which we regard as a point in the plane. Then any closed disk that contains all of 
the points 44; must contain an eigenvalue of A. 
[HW53] Let A and B be normal matrices with eigenvalues a ,--- ,@, and B),--- , By. Then 


n 


n 
B 2 2 2 
min) lai — Bowl? <A — Bile S max} [lay — Boils 
i=1 


oS, f=] 


where the minimum and maximum are over all permutations o in the symmetric group S, 
(i.e., the group of all permutations of 1,... ,7). 


[Sun82] [Bha82] Let A and B ben x n normal matrices with eigenvalues a ,--- ,@, and f1,--- 5 Bn. 
Let Ay, Ag be the diagonal matrices with diagonal entries a),--- ,a@, and B),--- , By, respectively. 
Let || - || be any unitarily invariant norm. Then, if A — B is normal, we have 


min |[A 4 — P~'AgP|| < |A— BI < max || A 4 — P~'AgP\l, 


where the maximum and minimum are over all n x m permutation matrices P. 

Observe that if A and B are Hermitian, then A — B is also Hermitian and, hence, normal, so 
this inequality holds for all pairs of Hermitian matrices. However, Example 6 gives a pair of 2 x 2 
normal matrices (with A — B not normal) for which the inequality does not hold. Note that for 
the Frobenius norm, we get the Hoffman—Wielandt inequality (20), which does hold for all pairs 
of normal matrices. 

For the operator norm, || - ||2, this gives the inequality 

rina; — Boy| S |A- Bll, s ee — Boj)! 
(assuming A — B is normal), which, for the case of Hermitian A and B, is a classical result of Weyl 
[Wey12]. 
[OS90] [BEK97][BDM83][BDK89][Hol92][AN86] Let A and B be normal matrices with eigen- 
values aj,--- ,@, and 6,--- , By, respectively. Using || All2 < ||Allz < || All2 together with the 
Hoffman—Wielandt inequality (20) yields 


1 : 
es, mip max let; — Boj)| < |A- Bll Ss vin max max |q — Bocj)l- 


Vi FESn 
On the right-hand side, the factor ,/n may be replaced by /2 and it is known that this constant is 


1 
the best possible. On the left-hand side, the factor —= may be replaced by the constant Te but 
Fr 
the best possible value for this constant is still unknown. Thus, we have 
li. 33 a 
Ty Dale — Bol < WA Bila S meee) — Bo(j)l- 


See also [Bha82], [Bha87], [BH85], [Sun82], [Sund82]. 
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If A and B are normal matrices, then AB = BA if and only if A and B have Property L. This was 
established for Hermitian matrices by Motzkin and Taussky [MT52] and then generalized to the 
normal case by Wiegmann [Wieg53]. For a stronger generalization see [Wiel53]. 


[Fri02] Let ajj,i = 1,...,n,j =1,...,n, be any set of meh) complex numbers. Then there 
exists an x n normal matrix, N, such that njj = aj; fori < j. Thus, any upper triangular matrix 
A can be completed to a normal matrix. 

[Bha87, p. 54] Let A be a normal n x n matrix and let B be an arbitrary n x n matrix such that 
|| A — Bllz < €. Then every eigenvalue of B is within distance € of an eigenvalue of A. Example 7 
below shows that this need not hold for an arbitrary pair of matrices. 

There are various ways to measure the “nonnormality” of a matrix. For example, if A has eigen- 


values 41,A2,... 5 An, the quantity Vl All} — 00, |Ail? is a natural measure of nonnormality, as 
is || A*A — AA*||2. One could also consider || A*A — AA*|| for other choices of norm, or look 
at min{||A — N|| : N is normal}. Fact 8 above suggests || HK — K H|| as a possible measure of 
nonnormality, while the polar decomposition (see Fact 7 above) A = UP of A suggests || UP— PU||. 
See [EP87] for more measures of nonnormality and comparisons between them. 

[Lin97] [FR96] For any € > 0 there isa > 0 such that, for any n x n complex matrix A with 
|| AA* — A*All. < 6, there is a normal matrix N with || N — All < ¢. Thus, a matrix which is 
approximately normal is close to a normal matrix. 


Examples: 


L's 


= Spelt 
. LetQ= i soe) Pte = we 


4 


a 
4 0 


0 
4 and A = 4P, + 2P, where the 


1 
mau = 7} | tn o-av = 


13 1 -l 


P/s are the pairwise orthogonal, orthogonal projection matrices 

OO) ae [fae Sg 
ea)? =e E |: 
1 44+21 6 1 2+1 


4 
A=|0 8+2i 0|]=H+iK,where H =/2-i 8 i} andK = ]1+2i 2 —1 
2 —2i 4i 4 -1 0 


1 0 


Pisa oo <6 


2/1 1 


1 
U* = — I i and Py =U 


are Hermitian. 


A= / and B= F | are both normal matrices, but the product AB = F i is 


1 0 
not normal. 


2 0 0 
Let A= |]0 O 1]. Then p(A) = 2 = |/All2, but A is not normal. 
0 0 0 


e? 


0 


cosO- sin@ 


i ol 


and D = | oa} The Q = UDU* = 


0  -i0 
eo 
ed =| ‘Al 


is a real, skew-symmetric 


6 0 
0 -6é 


0 
0 -é 


.|0 0 
u (ews |f 9 


Then H is Hermitian and Q = exp(iH). Also, K = iH = E 


Je — opi(u x 


u*). put = U 


matrix and Q = exp(K). 
Here is an example from [Sund82] showing that the condition that A — B be normal cannot be 


dropped from 21. Let A = 0 1 0 


i and B = f rl . Then A is Hermitian with eigenvalues +1 
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and B is skew-Hermitian with eigenvalues +i. So, we have ||A, — P~'AgP|l2 = /2, regardless 
: 2 
of the permutation P. However, A — B = : 0 and || A — Bll. = 2. 
7. This example shows that Fact 25 above does not hold for general pairs of matrices. Leta > B > 0 
0 a-fp 
0 0 


(og 


and put A = ; 0 


| awe = 


| . Then the eigenvalues of A are +,/aB and both 


0 £B 
B 0 


eigenvalues of B are zero. We have A — B = and ||A — B||, = B. But, sincea > B, we 


have /aB > B = ||A— BI). 
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8.1 Hermitian Matrices 


All matrices in this section are either real or complex, unless explicitly stated otherwise. 


Definitions: 

A matrix A ¢ C"*" is Hermitian or self-adjoint if A* = A, or element-wise, 4;; = aji,fori,j =1,...,n. 
The set of Hermitian matrices of order n is denoted by H,,. Note that a matrix A € R”"*” is Hermitian if 
and only if A? = A. 

A matrix A € C”*” is symmetric if A’ = A, or element-wise, aj; = a;i, fori, j = 1,...,n. The set of 
real symmetric matrices of order n is denoted by S,,. Since S, is a subset of 7, all theorems for matrices 
in H,, apply to S,, as well. 

Let V be a complex inner product space with inner product (v, w) and let v), v2,...,V, € V. The matrix 
G=l[gij] € C"*" defined by 8ij = (Vinvj), i,j € {1,2,...,} is called the Gram matrix of the vectors 
V1, V25.2-5 Vp 

The inner product (x, y) of two vectors x, y € C” will mean the standard inner product, ie., (x,y) = y*x, 
unless stated otherwise. The term orthogonal will mean orthogonal with respect to this inner product, 
unless stated otherwise. 


Facts: 


For facts without a specific reference, see [HJ85, pp. 38, 101-104, 169-171, 175], [Lax96, pp. 80-83], and 
[GRO1, pp. 169-171]. Many are an immediate consequence of the definition. 


1. A real symmetric matrix is Hermitian, and a real Hermitian matrix is symmetric. 
2. Let A, B be Hermitian. 
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(a) Then A + B is Hermitian. 

(b) If AB = BA, then AB is Hermitian. 

(c) Ifc € R, then cA is Hermitian. 

A+ A*, A* + A, AA*, and A*A are Hermitian for all A e C”””. 
If A € H,, then (Ax, y) = (x, Ay) for all x,y € C”. 

If A € H,, then A* € H, for allk € N. 

If A € H, is invertible, then A7! € Hy. 

The main diagonal entries of a Hermitian matrix are real. 


. All eigenvalues of a Hermitian matrix are real. 
. Eigenvectors corresponding to distinct eigenvalues of a Hermitian matrix are orthogonal. 
. Spectral Theorem — Diagonalization version: If A € H,, there is a unitary matrix U € C"*" 


such that U*AU = D, where D is a real diagonal matrix whose diagonal entries are the eigen- 
values of A. If A € S,, the same conclusion holds with an orthogonal matrix Q € R"*", 
ie, QTAQ =D. 

Spectral Theorem — Orthonormal basis version: If A € H,, there is an orthonormal basis of 
C" consisting of eigenvectors of A. If A € S,, the same conclusion holds with C” replaced 
by R". 

[Lay97, p. 447] Spectral Theorem — Sum of rank one projections version: Let A € H,, with eigenvalues 
Ay, A25..+5An, and corresponding orthonormal eigenvectors uj, U2,...,U,. Then 


A=A, uu; +A, uuy +--+ +A, uu. 


If A € S,, then 


A=h, uu +d, wu, +---+h, u,u). 
If A € H,, then rank A equals the number of nonzero eigenvalues of A. 
Each A € C”*" can be written uniquely as A = H + iK, where H, K € Hy. 
Given A € C"*", then A € H,, if and only if x* Ax is real for all x € C". 
Any Gram matrix is Hermitian. Some examples of how Gram matrices arise are given in Chapter 66 
and [Lax96, p. 124]. 
The properties given above for H,, and S,, are generally not true for symmetric matrices in C"””, 
but there is a substantial theory associated with them. (See [HJ85, sections 4.4 and 4.6].) 


Examples: 


1. 


2. 


: 6 0 2 
; 3 2- 
The matrix : f €Hy,and|0 —1 5] € 83. 
2+1 —5 > 5 3 


Let D be an open set in R” containing the point xo, and let f : D — R be a twice continu- 


a2 
ously differentiable function on D. Define H € R"*” by hij = 5 Eta.) Then H is a real 
Xj Xj 


symmetric matrix, and is called the Hessian of f. 


. Let G = (V,E) be a simple undirected graph with vertex set V = {1,2,3,...,n}. Then x n 


adjacency matrix A(G) = [aj;] (see Section 28.3) is defined by 


1 ifijek 
«=| if ij € 


0 otherwise. 


In particular, all diagonal entries of A(G) are 0. Since ij is an edge of G if and only if ji is, the 
adjacency matrix is real symmetric. Observe that for eachi € V, pas aj; = 4(1), Le., the sum of 
the i'” row is the degree of vertex i. 
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8.2 Order Properties of Eigenvalues of Hermitian Matrices 


Definitions: 


*A Ax, 
Given A € H,, the Rayleigh quotient R4 : C"\{0} > Ris Ra(x) = ena ae 


x*x (x, x) | 


Facts: 


For facts without a specific reference, see [HJ85, Sections 4.2, 4.3]; however, in that source the eigenvalues 
are labeled from smallest to greatest and the definition of majorizes (see Preliminaries) has a similar reversal 
of notation. 


1. Rayleigh-Ritz Theorem: Let A € H,, with eigenvalues A; > Az > +++ > An. 


Then 
x 
x* Ax 
An S <2, forall nonzero x € C”, 
x* Ax ‘ 
A, = max = max x’ Ax, 
x40 X*X Ix|]2=1 
and 
. x*Ax tte 
dA, = min = min x’ Ax. 
x40 X*x \x\]2=1 


2. Courant—Fischer Theorem: Let A € M,, bea Hermitian matrix with eigenvaluesA,; > Az >... > An 
and let k be a given integer with 1 < k < n. Then 


. x* Ax 
max min = = Xk 
n a 
W15W2).--) Wn—k eC xF 0xeC x"x 
X L W1,Wo)..-) Wik 


and 


: x* Ax 
min max iS Xx. 
a a 
W],W25.--) Wk—-1 eC x#0,xeC x’x 
XL W1,W2,..5 Whi 


3. (Also [Bha0l, p. 291]) Weyl Inequalities: Let A, B € H,, and assume that the eigenvalues of A, B 
and A+ B are arranged in decreasing order. Then for every pair ofintegers j,k suchthat1 < j,k <n 
andj +k<n+l, 


Aj+k-1(A + B) < Aj(A) + Ax(B) 
and for every pair of integers j,k such that 1 < j,k <nandj+k>n+1, 
Aj+k—-n(A + B) = Aj (A) + Ax(B). 


4. Weyl Inequalities: These inequalities are a prominent special case of Fact 3. Let A,B € H,, and 
assume that the eigenvalues of A, B and A+ B are arranged in decreasing order. Then for each 
j € {1,2,...,n}, 


Aj (A) + An(B) <Aj(A+ B) <Aj(A) + A1(B). 


5. Interlacing Inequalities: Let A € H,, let Ay > Ay > --- > A, be the eigenvalues of A, and for any 
i € {1,2,...,n}, let wy > po > +--+ > fy_) be the eigenvalues of A(z), where A(z) is the principal 
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submatrix of A obtained by deleting its i” row and column. Then 


Ay = My 2 Ad = Mg SAB See SAW = Mn-1 = An- 


Let A € H, and let B be any principal submatrix of A. If A; is the k"” largest eigenvalue of A and 
pg is the kt largest eigenvalue of B, then A, > pg. 

Let A € H,, with eigenvalues A} > Az > +++ > An. Let S be a k-dimensional subspace of C” with 
k € {1,2,...,n}. Then 


(a) Ifthere is a constant c such that x* Ax > cx*x for allx € S, then A, > c. 
(b) If there is a constant c such that x* Ax < cx*x for all x € S, then Ay,_441 <c. 


Let A € Hy. 


(a) If x* Ax > 0 for all x in a k-dimensional subspace of C”, then A has at least k nonnegative 
eigenvalues. 

(b) Ifx* Ax > 0 for all nonzero x in a k-dimensional subspace of C", then A has at least k positive 
eigenvalues. 


Let A € H,, let A = (Aj,A2,...,An) be the vector of eigenvalues of A arranged in decreasing 
order, and let a = (a), 42,...,a,) be the vector consisting of the diagonal entries of A arranged in 
decreasing order. Then A > aq. (See Preliminaries for the definition of >.) 

Let @ = (a),42,...;4n), B = (bi, b2,...,b,) be decreasing sequences of real numbers such that 
a > f. Then there exists an A € H,, such that the eigenvalues of A are aj, a2,...,a,, and the 
diagonal entries of A are bj, b2,..., Dn. 

[Lax96, pp. 133-6] or [Bha01, p. 291] (See also Chapter 15.) Let A,B € H, with eigenvalues 
A\(A) = A2(A) = +++ B An(A) and Aj (B) > A2(B) = +--+ = An(B). Then 


(a) |A;(A) —A;(B)| < ||A— Bllo, i =1,...,n. 
(b) > [a;(A) — A;(B)])? < ||A — Bip. 
i=1 


Examples: 


1. 


25 


Setting x = e; in the Rayleigh-Ritz theorem, we obtain A,, < aj; < 41. Thus, for any A € H,, we 
have A, > max{a;;|i € {1,2,...,n}} andd, < minf{a;;|i € {1,2,...,n}}. 

Setting x = [1,1,..., 1] in the Rayleigh-Ritz theorem, we find that 4, < i ei aij < A,. Ifwe 
take A to be the adjacency matrix ofa graph, then this inequality implies that the largest eigenvalue 
of the graph is greater than or equal to its average degree. 

The Weyl inequalities in Fact 3 above are a special case of the following general class of inequalities: 


So an(A + B) < $5 Ai(A) + 55 j(B), 


keK ieI joJ 


where I, J, K are certain subsets of {1,2,...,}. In 1962, A. Horn conjectured which inequalities 
of this form are valid for all Hermitian A, B, and this conjecture was proved correct in papers by 
A. Klyachko in 1998 and by A. Knutson and T. Tao in 1999. Two detailed accounts of the problem 
and its solution are given in [Bha01] and [Ful00]. 


1 1 1 0 
1 1 1 0 : : 
Let A = ie A have eigenvalues 4; > Ap > A3 > Ag. Since A(4) = ]1 1 1 
1 


0 0 1 1 
has eigenvalues 3, 0, 0, by the interlacing inequalities, A; > 3 > A. > 0 >A3 > 0 > Ag. In 
particular, A3 = 0. 
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Applications: 


1. 


8.3 


To use the Rayleigh—Ritz theorem effectively to estimate the largest or smallest eigenvalue of a 
Hermitian matrix, one needs to take into account the relative magnitudes of the entries of the 


PD. 
matrix. For example, let A= |1 2 2]. In order to estimate A), we should try to maximize the 
Ie -2: 3 


Rayleigh quotient. A vector x € R? is needed for which no component is zero, but such that each 
component is weighted more than the last. In a few trials, one is led to x = [1, 2, 3] T which gives a 


1 Af 
Rayleigh quotient of 5. So A, > 5. This is close to the actual value of 4, which is % csc? ri = 5.049. 


This example is only meant to illustrate the method; its primary importance is as a tool for 
estimating the largest (smallest) eigenvalue of a large Hermitian matrix when it can neither be 
found exactly nor be computed numerically. 

The interlacing inequalities can sometimes be used to efficiently find all the eigenvalues of a Her- 
mitian matrix. The Laplacian matrix (from spectral graph theory, see Section 28.4) of a star is 


Woe SPS ae Sa 
= rr ce 0 0 
= ee ae 0 0 
L= 
=e 1 0 
St. 0" IG v3? Ok «3 


Since L(1) is an identity matrix, the interlacing inequalities relative to L(1) are: Ay > 1 > 2 > 
1>...>Ay-1 => 1 > Ay. Therefore, n — 2 of the eigenvalues of L are equal to 1. Since the columns 
sum to 0, another eigenvalue is 0. Finally, since tr L = 2n — 2, the remaining eigenvalue is n. 


. The sixth fact above is applied in spectral graph theory to establish the useful fact that the k’ 


largest eigenvalue of a graph is greater than or equal to the k‘” largest eigenvalue of any induced 
subgraph. 


Congruence 


Definitions: 


Two matrices A, B € H,, are *congruent if there is an invertible matrix C € C"*" such that B = C* AC, 


denoted A ~ B.IfC is real, then A and B are also called congruent. 

Let A € H,. The inertia of A is the ordered triple in(A) = (a (A), v(A),6(A)), where (A) is the 
number of positive eigenvalues of A, v(A) is the number of negative eigenvalues of A, and 5(A) is the 
number of zero eigenvalues of A. 

In the event that A € C”™” has all real eigenvalues, we adopt the same definition for in(A). 


Facts: 


The following can be found in [HJ85, pp. 221-223] and a variation of the last in [Lax96, pp. 77-78]. 


Ot ee 


Unitary similarity is a special case of “congruence. 

*Congruence is an equivalence relation. 

For A € Hy, 1(A) + v(A) + 6(A) = 1. 

For A € H,, rank A = m(A) + v(A). 

Let A € H,, with inertia (r,s,t). Then A is *congruent to I, @ (—I;) © 0;. A matrix C that 
implements this *congruence is found as follows. Let U be a unitary matrix for which U* AU = D 


8-6 Handbook of Linear Algebra 


is a diagonal matrix with dj;,...,d,, the positive eigenvalues, d,+1,+41,---5drisy+s the negative 
eigenvalues, and dj; = 0, k > r +s. Let 


1// dij, ol 
s§=41//—-dj;, i=r4+l,...,s 


1, it>rcts 


and let S = diag(s),52,...,5,). Then C = US. 


6. Sylvester’s Law of Inertia: Two matrices A, B € 1, are *congruent if and only if they have the same 
inertia. 


Examples: 
0 0 3 
1. LettA=|0 0 4]. Since rank A = 2, 7(A) + v(A) = 2, so 6(A) = 1. Since tr A = 0, we have 
3 4 0 


w(A) = v(A) = 1, and in(A) = (1, 1, 1). Letting 


3 3 4 
5/10 54/10 5 
C 4 4 3 
™~ 1 54/10 5/10 5 
oe es ee | 
/10 J10 
we have 
1 0 O 
C*AC=|0 —-1 0 
0 0 O 
Now suppose 
0 1 0 
B=]/{1 0 1 
0 1 0 


Clearly in(B) = (1, 1, 1) also. By Sylvester’s law of inertia, B must be *congruent to A. 


8.4 Positive Definite Matrices 


Definitions: 


A matrix A € H,, is positive definite if x* Ax > 0 for all nonzero x € C”. It is positive semidefinite if 
x* Ax > 0 for all x € C”. It is indefinite if neither A nor —A is positive semidefinite. The set of positive 
definite matrices of order n is denoted by PD,, and the set of positive semidefinite matrices of order n 
by PSD, If the dependence on n is not significant, these can be abbreviated as PD and PSD. Finally, PD 
(PSD) are also used to abbreviate “positive definite” (“positive semidefinite”). 

Let k be a positive integer. If A, B are PSD and B* = A, then B is called a PSD k‘" root of A and is 
denoted A!/*, 


A correlation matrix is a PSD matrix in which every main diagonal entry is 1. 
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Facts: 


For facts without a specific reference, see [HJ85, Sections 7.1 and 7.2] and [Fie86, pp. 51-57]. 


wo ON DAM 


10. 


11. 
12. 


13. 
14. 
15. 
16. 


17. 
18. 


19. 


20. 


21. 


22. 


A€S, is PD ifx! Ax > 0 for all nonzero x € R", and is PSD if x? Ax > 0 for allx € R”. 


. Let A, B € PSDy. 


(a) Then A+ B € PSDy. 

(b) If in addition, A € PD,, then A+ B € PD,. 

(c) Ifc > 0, thenc A € PSD,. 

(d) If, in addition, A € PD, andc > 0, thenc A € PD,. 


. If Ay, Ao,..., Ap € PSD,, then so is A} + Az +---+ Ax. If, in addition, there is ani € {1,2,...,k} 


such that A; € PD,,, then A; + Ap +--+: + Ag € PDy. 


. Let A € H,. Then A is PD if and only if every eigenvalue of A is positive, and A is PSD if and only 


if every eigenvalue of A is nonnegative. 


. If Ais PD, then tr A > 0 and det A > 0. If A is PSD, then tr A > 0 and det A > 0. 


A PSD matrix is PD if and only if it is invertible. 


. Inheritance Principle: Any principal submatrix of a PD (PSD) matrix is PD (PSD). 
. All principal minors of a PD (PSD) matrix are positive (nonnegative). 
. Each diagonal entry of a PD (PSD) matrix is positive (nonnegative). If a diagonal entry of a PSD 


matrix is 0, then every entry in the row and column containing it is also 0. 
Let A € H,. Then A is PD if and only if every leading principal minor of A is positive. A is PSD 


: 3 ae : ; ; . |O O oe 
if and only if every principal minor of A is nonnegative. (The matrix | — , shows that it is not 


sufficient that every leading principal minor be nonnegative in order for A to be PSD.) 

Let A be PD (PSD). Then A‘ is PD (PSD) for all k € N. 

Let A € PSD, and express A as A = UDU*, where U is unitary and D is the diagonal matrix 
of eigenvalues. Given any positive integer k, there exists a unique PSD k’" root of A given by 
Ak — UD'‘U*, If A is real so is A'/*, (See also Chapter 11.2.) 

If A is PD, then A~! is PD. 

Let A € PSD,, and let C € C”*". Then C* AC is PSD. 

Let A € PD, and let C ¢ C”*”, n > m. Then C* AC is PD if and only if rank C = m; ie., if and 
only if C has linearly independent columns. 

Let A € PD, and C € C"*”. Then C* AC is PD if and only if C is invertible. 

Let A € H,. Then A is PD if and only if there is an invertible B € C"*” such that A = B*B. 
Cholesky Factorization: Let A € H,. Then A is PD ifand only if there is an invertible lower triangular 
matrix L with positive diagonal entries such that A = LL*. (See Chapter 38 for information on 
the computation of the Cholesky factorization.) 

Let A € PSD, with rank A = r < n. Then A can be factored as A = B*B with B € C’™". If 
A is a real matrix, then B can be taken to be real and A = B’B. Equivalently, there exist vectors 
Vis V2y--29Vn € C’ (or R’) such that a;; = viv; (or v/v;). Note that A is the Gram matrix (see 
section 8.1) of the vectors vj, V2,...,V,. In particular, any rank 1 PSD matrix has the form xx* for 
some nonzero vector x € C", 

[Lax96, p. 123]; see also [HJ85, p. 407] The Gram matrix G ofa set of vectors V1, V2,...,Vn is PSD. 
If Vv), V2,...;,V, are linearly independent, then G is PD. 

[HJ85, p. 412] Polar Form: Let A € C”*",m > n. Then A can be factored A = UP, where 
P € PSD, rank P = rank A, and U € C”*" has orthonormal columns. Moreover, P is uniquely 
determined by A and equals (A*A)!/?_ If A is real, then P and U are real. (See also Section 17.1.) 
[HJ85, p. 400] Any matrix A € PD, is diagonally congruent to a correlation matrix via the diagonal 


matrix D = (1/,/aq1,.-.51/,/@nn). 
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23. [BJT93] Parameterization of Correlation Matrices in S3: Let 0 < a, B, y < a. Then the matrix 
1 cosa cosy 
C= |cosa 1 cos B 
cosy cosp 1 
is PSDifandonlyifa<B+y, Baty, y<at+ f, a+f8+y < 2m. Furthermore, C 
is PD if and only if all of these inequalities are strict. 
24. [HJ85, p. 472] and [Fie86, p. 55] Let A = 2 : € H,, and assume that B is invertible. Then 
Ais PD if and only if the matrices B and its Schur complement S = D — C*B7!C are PD. 
25. [Joh92] and [LB96, pp. 93-94] Let A = " be PSD. Then any column of C lies in the span 
of the columns of B. 
26. [HJ85, p. 465] Let A € PD, and B € H,,. Then 
(a) AB is diagonalizable. 
(b) All eigenvalues of AB are real. 
(c) in(AB) = in(B). 
27. Any diagonalizable matrix A with real eigenvalues can be factored as A = BC, where B is PSD and 
C is Hermitian. 
28. If A, B € PD,, then every eigenvalue of AB is positive. 
29. [Lax96, p. 120] Let A, B € H,. If Ais PD and AB + BA is PD, then B is PD. It is not true that if 
A, B are both PD, then AB + BA is PD as can be seen by the example A = : : , B= ; i F 
30. [HJ85, pp. 466-467] and [Lax96, pp. 125-126] The real valued function f(X) = log(det X) is 
concave on the set PD,; ie. f((l -—t)X +tY) > (1—t) f(X)+tf(Y) for all t € [0,1] and all 
X,Y € PDy. 
31. [Lax96, p. 129] If A € PDy is real [ “al he gy = TE 
. [Lax96, p. n is real, e x= : 
F R’ Vv det A 
32. [Fie60] Let A = [aij], B= [bij] € PD,, with At= [aij], Bol= [Bij]. Then 
S- (aij — bij (aig — Bij) < 0, 
ij=l 
with equality if and only if A = B. 
2 v 
33. [Ber73, p. 55] Consider PD,, to beasubset of C” (or for real matrices of R” ). Then the (topological) 
boundary of PD,, is PSD,,. 
Examples: 
1. If A = [a] is 1 x 1, then A is PD if and only if a > 0, and is PSD if and only if a > 0; so PD and 
PSD matrices are a generalization of positive numbers and nonnegative numbers. 
2. If one attempts to define PD (or PSD) for nonsymmetric real matrices according to the the usual 


definition, many of the facts above for (Hermitian) PD matrices no longer hold. For example, 


suppose A = 


. Then x? Ax = 0 for all x € R’. But o (A) = {i, —i}, which does not agree 


with Fact 4 above. 
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ong x (hte Te a] 25 0) 1 Liat. ol on 
3. The matrix A = E "| factors = [I | i Pal {i} A = 


ER se Ol) eae |) Pa 
J2\|1 —1/}0 3) f2|1 —-1]) |i 4° 


4. A self-adjoint linear operator on a complex inner product space V (see Section 5.3) is called positive 
if (Ax, x) > 0 for all nonzero x € V. For the usual inner product in C” we have (Ax, x) = x* Ax, 
in which case the definition of positive operator and positive definite matrix coincide. 

5. Let X), X2,...,X, be real-valued random variables on a probability space, each with mean zero 
and finite second moment. Define the matrix 


aij = E(X;X;), i,j € {1,2,...,n}. 


The real symmetric matrix A is called the covariance matrix of X), X2,..., X,, and is necessarily 
PSD. If we let X = (X,, X2,..., Xn)7, then we may abbreviate the definition to A = E(XX7‘). 


Applications: 


1. [HFKLMO95, p. 181] or [MT88, p. 253] Test for Maxima and Minima in Several Variables: Let D 
be an open set in R” containing the point xo, let f : D > R bea twice continuously differentiable 
function on D, and assume that all first derivatives of f vanish at xp. Let H be the Hessian matrix 
of f (Example 2 of Section 8.1). Then 


(a) f has a relative minimum at xp if H(xo) is PD. 
(b) f has a relative maximum at Xo if — H(xo) is PD. 
(c) f has a saddle point at xp if H(xo) is indefinite. 


Otherwise, the test is inconclusive. 

2. Section 1.3 of the textbook [Str86] is an elementary introduction to real PD matrices empha- 
sizing the significance of the Cholesky-like factorization LDL’ of a PD matrix. This represen- 
tation is then used as a framework for many applications throughout the first three chapters of 
this text. 

3. Let A bea real matrix in PD,,. A multivariate normal distribution is one whose probability density 
function in R” is given by 


= il =1xTAIx 
fx)= SG : 


It follows from Fact 31 above that JR" f(x) dx = 1. A Gaussian family X,, X2,...X,, where 
each X; has mean zero, is a set of random variables that have a multivariate normal distribution. 
The entries of the matrix A satisfy the identity aj; = E(X;X/j), so the distribution is completely 
determined by its covariance matrix. 


8.5 Further Topics in Positive Definite Matrices 


Definitions: 
Let A,B € F"*", where F is a field. The Hadamard product or Schur product of A and B, denoted 
Ao B, is the matrix in F"*" whose (i, a entry is aj; b;;. 

A function f :R — C is called positive semidefinite if for each n € N and all x1, x,...,x, € R, the 
n x n matrix [ f(x; — x;)] is PSD. 


8-10 Handbook of Linear Algebra 


Let A, B € H,. We write A > B if A— Bis PD, and A > B if A— B is PSD. The partial ordering on 
H, induced by > is called the partial semidefinite ordering or the Loewner ordering. 
Let V be an n-dimensional inner product space over C or R. A set K C V is called a cone if 


(a) Foreachx,ye K,x+yeK. 
(b) Ifx € K andc > 0,thencx € K. 


A cone is frequently referred to as a convex cone. A cone K is closed if K is a closed subset of V, is 
pointed if K 1 —K = {0}, and is full if it has a nonempty interior. The set 


K*={yeV|(xy)>0 VxeK} 


is called the dual space. 


Facts: 


1. [HJ91, pp. 308-309]; also see [HJ85, p. 458] or [Lax96, pp. 124, 234] Schur Product Theorem: If 
A, B € PSD,, then so is Ao B. If A € PSD,, aj, > 0, i = 1,...,n, and B € PD,, then Ao Be 
PD,,. In particular, if A and B are both PD, then so is Ao B. 

2. [HJ85, p. 459] Fejer’s Theorem: Let A = [ajj] € H,. Then A is PSD if and only if 


ye Gij bij > 0 
i,j=l 
for all matrices B € PSD,. 

3. [HJ91, pp. 245-246] If A € PD,, and B ¢€ PD,, then the Kronecker (tensor) product (see 
Section 10.4) A @ B € PDinn. If A € PSD,, and B € PSD,, then A ® B € PSDinn. 

4. [HJ85, p. 477] or [Lax96, pp. 126-127, 131-132] Hadamard’s Determinantal Inequality: If A € PD, 
then det A < []j_, a;;. Equality holds if and only if A is a diagonal matrix. 

5. [FJ00, pp. 199-200] or [HJ85, p. 478] Fischer’s Determinantal Inequality: If A € PD, and q@ is any 
subset of {1,2,...,n}, then det A < det A[a] det A[a‘] (where det A[#] = 1). Equality occurs if 
and only if A[a@, a] is a zero matrix. (See Chapter 1.2 for the definition of A[a] and A[qa, f].) 

6. [FJ00, pp. 199-200] or [HJ85, p. 485] Koteljanskii’s Determinantal Inequality: Let A € PD,, and let 
a, B be any subsets of {1,2,...,n}. Then det Ala U B] det A[a NM B] < det A[a] det A[B]. Note 
that if N B = Y, Koteljanskii’s inequality reduces to Fischer’s inequality. Koteljanskii’s inequality 
is also called the Hadamard—Fischer inequality. 

For other determinantal inequalities for PD matrices, see [FJ00] and [HJ85, 87.8]. 

7. [Fel71, pp. 620-623] and [Rud62, pp. 19-21] Bochner’s Theorem: A continuous function from R 
into C is positive semidefinite if and only if it is the Fourier transform of a finite positive measure. 

8. [Lax96, p. 118] and [HJ85, p. 475, 470] Let A, B,C, D € Hy. 


(a) If A< BandC < D,thenA+C < B+D. 
(b) If A <x Band B < C, then A < C. 
(c) If A ~ Band S € C”*" is invertible, then S* AS ~ S*BS. 


The three statements obtained by replacing each occurrence of < by < are also valid. 
9. [Lax96, pp. 118-119, 121-122] and [HJ85, pp. 471-472] Let A, B € PD,, with A < B. Then 


(a) Atl > Bol. 
(b) Al? ~ BY?, 
(c) det A < det B. 
(d) trA <tr B. 


If A = B, then statement (a) holds with > replaced by >, statement (b) holds with < replaced by 
<, and statements (c) and (d) hold with < replaced by <. 
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10. [HJ85, pp. 182, 471-472] Let A,B € H,, with eigenvalues 11(A) > A2(A) > --- > A,(A) and 
A1(B) > A2(B) > --- > A,(B). If A ~ B, then 4,(A) < Ax(B), k = 1,...,n. If A < B, then 
An (A) < Axg(B), k= 1,...,n. 

11. [HJ85, p. 474] Let A be PD and let w C {1,2,...,n}. Then A~![a@] > (A[a])7!. 

12. [HJ85, p. 475] If Ais PD, then Ato A> I > (A710 A). 

13. [Hal83, p. 89] If K isa cone in an inner product space V, its dual space is a closed cone and is called 
the dual cone of K. If K is a closed cone, then (K*)* = K. 

14. [Ber73, pp. 49-50, 55] and [HW87, p. 82] For each pair A, B € Hy, define (A, B) = tr (AB). 


(a) H, is an inner product space over the real numbers with respect to (-,-). 
(b) PSD, is a closed, pointed, full cone in Hy. 
(c) (PSD,,)* = PSD,. 


Examples: 


1. The matrix C = [cos|i — j|] € S, is PSD, as can be verified with Fact 19 of section 8.4 and 
the addition formula for the cosine. But a quick way to see it is to consider the measure (x) = 
$[5(x +1)4+ 6(x — 1l)jsie, w(E) = Oif—1,1 ¢ E, u(E) = 1if—1,1 € FE, and w(E£) = 1/2 
if exactly one of —1,1 € E. Since the Fourier transform of yu is cost, if we let x1, x2,...,X, be 
1,2,... , min the definition of positive definite function, we see immediately by Bochner’s Theorem 
that the matrix [cos(i — j)] = [cos |i — j|] = C is PSD. By Hadamard’s determinantal inequality 
detC < Tia (eae) 


fra a) ea 2-1 
2. Since I al ; 5] kings we ave see “ape E al 
i Dea 2 -1 0 
3. The matrix A= |1 2 2] is PDwithinverse AT’ = |—1 2 —1)]. Then (A[{1,3}])7! = 
Ble rf, 
o S19 0! 0 
15. =.5 2 0 
i. | 7 E i = A™'[{1,3}]. Also, Ato A= |-1 4 =2) = |0 1 0} = 
. 0. 8=2 3 0 0 1 
8 3 2 
413 6 4) =(Ato A)! 
2 4 7 


: . 2 1 1 0 
4. If A > B > 0, it does not follow that A* > B*. For example, if A = 11 and B = | ; 


then B and A — B are PSD, but A? — B? is not. 


Applications: 


1. Hadamard’s determinantal inequality can be used to obtain a sharp bound on the determinant of 
a matrix in C"*" if only the magnitudes of the entries are known. [HJ85, pp. 477-478] or [Lax96, 
p. 127]. 

Hadamard’s Determinantal Inequality for Matrices in C"*": Let B € C"*". Then | det B| < 
Tj Oo jar Ii |’)? with equality holding if and only if the rows of B are orthogonal. 

In the case that B is invertible, the inequality follows from Hadamard’s determinantal inequality 
for positive definite matrices by using A = B B*; if B is singular, the inequality is obvious. 

The inequality can be alternatively expressed as | det B| < ]]}_, ||b;||2, where b; are the rows of 
B. If B isa real matrix, it has the geometric meaning that among all parallelepipeds with given side 
lengths ||b;||2, 7 = 1,...,7, the one with the largest volume is rectangular. 

There is a corresponding inequality in which the right-hand side is the product of the lengths of 
the columns of B. 
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2. [Fel71, pp. 620-623] A special case of Bochner’s theorem, important in probability theory, is: A 
continuous function @ is the characteristic function of a probability distribution if and only if it is 
positive semidefinite and #(0) = 1. 

3. Understanding the cone PSD,, is important in semidefinite programming. (See Chapter 51.) 
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Nonnegativity is a natural property of many measured quantities (physical and virtual). Consequently, non- 
negative matrices arise in modelling transformations in numerous branches of science and 
engineering — these include probability theory (Markov chains), population models, iterative methods in 
numerical analysis, economics (input-output models), epidemiology, statistical mechanics, stability anal- 
ysis, and physics. This section is concerned with properties of such matrices. The theory of the subject was 
originated in the pioneering work of Perron and Frobenius in [Per07a,Per07b,Fro08,Fro09, and Fro12]. 
There have been books, chapters in books, and hundreds of papers on the subject (e.g., [BNS89], [BP94], 
[Gan59, Chap. XII], [Har02] [HJ85, Chap. 8], [LT85, Chap. 15], [Min88], [Sen81], [Var62, Chap. 1]). A 
brief outline of proofs of the classic result of Perron and a description of several applications of the theory 
can be found in the survey paper [Mac00]. Generalizations of many facts reported herein to cone-invariant 
matrices can be found in Chapter 26. 


9.1 Notation, Terminology, and Preliminaries 


Definitions: 
For a positive integer n, (n) = {1,...,n}. 
For a matrix A € C”*": 


A is nonnegative (positive), written A > 0 (A > 0), ifall of A’s elements are nonnegative (positive). 


9-1 


9-2, Handbook of Linear Algebra 


A is semipositive, written A > 0if A> OandA+#0. 
|A| will denote the nonnegative matrix obtained by taking element-wise absolute values of A’s 
coordinates. 


For a square matrix A = [a;;] € C"*": 

The k-eigenspace of A at a complex number A, denoted Né(A), is ker(A — 41)*; a generalized eigen- 
vector of P at A is a vector in UZ.) N*(A). 

The index of A at A, denoted v,(A), is the smallest integer k with Nk(A) — NK*1(A). 

The ergodicity coefficient of A, denoted t(A), is max{|A|:A € o(A) and |A| 4 p(A)} (with the 
maximum over the empty set defined to be 0 and p(A) being the spectral radius of A). 

A group inverse of a square matrix A, denoted A*, is a matrix X satisfying AXA = A, XAX = X, and 
AX = XA (whenever there exists such an X, it is unique). 

The digraph of A, denoted I'(A), is the graph with vertex-set V(A) = (m) and arc-set E(A) = {(i, j) : 
i, j € (n) and a;; A 0}; in particular, i = 1,...,n are called vertices. 

Vertex i € (n) has access to vertex j € (), writteni +> j, if either i = j or (A) contains a simple 
walk (path) from i to j; we say that i and j communicate, written i ~ j, if each has access to the other. 

A subset C of (n) is final if no vertex in C has access to a vertex not in C. 

Vertex-communication is an equivalence relation. It partitions (m) into equivalence classes, called the 
access equivalence classes of A. 

I'(A) is strongly connected if there is only one access equivalence class. 

An access equivalence class C has access to an access equivalence class C’, written C > C’ if some, or 
equivalently every, vertex in C has access to some, or equivalently every, vertex in C’; in this case we also 
write i +> C’ and C b i’ wheni € C andi’ € C’. 

An access equivalence class C of A is final if its final as a subset of (n), that is, it does not have access to 
any access equivalence class but itself. 

The reduced digraph of (A), denoted R[I(A)], is the digraph whose vertex-set is the set of access 
equivalence classes of A and whose arcs are the pairs (C, C’) with C and C’ as distinct classes satisfying 
CRC’. 


For a sequence {4;1}:m=0,1,... of complex numbers and a complex number a: 

a isa (C,0)-limit of {4,,}=0,1,.... Written lim, dm = a (C,0), if lim—+0 4m = a (in the sense of a 
regular limit). 

a is the (C, 1)-limit of {4} n=0,1,.... Written lim oo dm = a (C, 1), iflimmy soo m7 ey ad,=a. 

Inductively for k = 2,3,..., a is a (C,k)-limit of {a} m=0,,.... written lim,,o04m = a (C,k), if 
lity votre ae = @ (Ck = 1); 


For0 < B < 1, {am} m=0,1,... converges geometrically to a with (geometric) rate 6 ifforeach B < y <1, 
ama 


the set of real numbers { at mn = 0,1,...} is bounded. (For simplicity, we avoid the reference of 
geometric convergence for (C, k)-limits.) 


For a square nonnegative matrix P: 

p(P) (the spectral radius of P) is called the Perron value of P (see Facts 9.2-1(b) and 9.2-5(a) and 
9.3-2(a)). 

A distinguished eigenvalue of P is a (necessarily nonnegative) eigenvalue of P that is associated with 
a semipositive (right) eigenvector. 

For more information about generalized eigenvectors, see Chapter 6.1. An example illustrating 
the digraph definitions is given in Figure 9.1; additional information about digraphs can be found in 
Chapter 29. 


9.2 Irreducible Matrices 


(See Chapter 27.3, Chapter 29.5, and Chapter 29.6 for additional information.) 
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Definitions: 
A nonnegative square matrix P is irreducible if it is not permutation similar to any matrix having the 


(nontrivial) block-partition 


A B 
0 C 


with A and C square. 

The period of an irreducible nonnegative square matrix P (also known as the index of imprimitivity 
of P) is the greatest common divisor of lengths of the cycles of '(P), the digraph of P. 

An irreducible nonnegative square matrix P is aperiodic if its period is 1. 
Note: We exclude from further consideration the (irreducible) trivial 0 matrix of dimension 1 x 1. 


Facts: 


Facts requiring proofs for which no specific reference is given can be found in [BP94, Chap. 2]. 


1. (Positive Matrices — Perron’s Theorem) [Per07a, Per07b] Let P be a positive square matrix with 
spectral radius p and ergodicity coefficient tT. 


(a) P is irreducible and aperiodic. 
(b) p is positive and is a simple eigenvalue of P; in particular, the index of P at p is 1. 


(c) There exist positive right and left eigenvectors of P corresponding to p, in particular, ¢ is a 
distinguished eigenvalue of both P and P’. 


(d) p is the only distinguished eigenvalue of P. 
(e) p is the only eigenvalue A of P with |A| = p. 
(f) Ifx € R" satisfies x > 0 and either (oI — P)x > Oor (pI — P)x < 0, then (oI — P)x = 0. 


(g) Ifvand ware positive right and left eigenvectors of P corresponding to p (note that w is a row 
vector), then limin-+00( 2)” = \* and the convergence is geometric with rate 3 


(h) Q = pI — P hasa group inverse; further, if v and ware positive right and left eigenvectors of P 
corresponding to p, then Q+ * is nonsingular, Qi = (Q+2)" (I-™), and = 1 QQ. 


(i) limy)-s00 ear : ‘i -~m* = (pI P)* and the convergence is geometric with rate o 
2. (Characterizing Irreducibility) Let P be a nonnegative n x n matrix with spectral radius p. The 


following are equivalent: 


(a) P is irreducible. 

(hy See PF 0; 

(c) (I+ P)*!>0. 

(d) The digraph of P is strongly connected, i.e., P has a single access equivalence class. 
(e) Every eigenvector of P corresponding to p is a scalar multiple of a positive vector. 
(f) For some x > p, I — P is nonsingular and (uI — P)~! > 0. 

(g) For every 4 > p, “I — P is nonsingular and («I — P)~! > 0. 


3. (Characterizing Aperiodicity) Let P be an irreducible nonnegative n x n matrix. The following are 
equivalent: 
(a) P is aperiodic. 
(b) P” > 0 for some m. (See Section 29.6.) 
(c) P™ > Oforallm>n. 
4. (The Period) Let P be an irreducible nonnegative n x n matrix with period q. 


(a) q is the greatest common divisor of {m:m is a positive integer and (P™);; > 0} for any one, or 
equivalently all, i € {1,...,n}. 
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(b) There exists a partition C),...,C, of {1,..., 1} such that: 


i. Fors,t =1,...,q, P[C;,C,;] 4 0 ifand only ift = s + 1 (with q + 1 identified with 1); in 
particular, P is permutation similar to a block rectangular matrix having a representation 


0 P[C,, Ca] 0 St 0 

0 0 PIG WC) 23 0 

0 0 0 ois, LCase 
P[Cq,C1] 0 0 fey 0 


ii. P4[C,] is irreducible fors = 1,...,q and P4[C,,C,] = Ofors,t=1,...,n withs # t;in 
particular, P4 is permutation similar to a block diagonal matrix having irreducible blocks 
on the diagonal. 


. (Spectral Properties — The Perron—Frobenius Theorem) |Fro12] Let P be an irreducible nonnegative 


square matrix with spectral radius p and period q. 


(a) p is positive and is a simple eigenvalue of P; in particular, the index of P at p is 1. 

(b) There exist positive right and left eigenvectors of P corresponding to p; in particular, ¢ is a 
distinguished eigenvalue of both P and P?. 

(c) p is the only distinguished eigenvalue of P and of P’. 

(d) Ifx € R” satisfies x > 0 and either (oI — P)x > 0 or (oI — P)x < 0, then (pI — P)x=0. 

(e) The eigenvalues of P with modulus p are {pe?7)*/4 :k = 0,...,q — 1} (here, i is the complex 
root of —1) and each of these eigenvalues is simple. In particular, if P is aperiodic (q = 1), 
then every eigenvalue 1 4 p¢ of P satisfies |A| < p. 

(f) Q = pI —P hasa group inverse; further, if v and w are positive right and left eigenvectors of P 
corresponding to p, then Q+ is nonsingular, Q* = (Q+™)"'(I—), and * = I- QQ. 


. (Convergence Properties of Powers) Let P be an irreducible nonnegative square matrix with spectral 


radius p, index v, period q, and ergodicity coefficient t. Also, let v and w be positive right and 


VW 


left eigenvectors of P corresponding to p and let P* = * 

(a) limn+00( 5) = P* (G1). 

(b) limp oo 5 ae at . )' = P* and the convergence is geometric with rate . < 1.Inparticular, 
if P is aperiodic (q = 1), then limm-+oo( 2)” = P* and the convergence is geometric with rate 


Eo det dr 
p 


(c) For eachk = 0,...,q —1, Limin +00( 2 )™t* exists and the convergence of these sequences to 
their limit is geometric with rate ( 5 \a<l, 

Clair Reema Beary - )' — mP* = (I — p | P)* (C,1); further, if P is aperiodic, this limit holds 
as a regular limit and the convergence is geometric with rate 5 < 1. 


. (Bounds on the Perron Value) Let P be an irreducible nonnegative n x n matrix with spectral radius 


p, let w be a nonnegative scalar, and let o € {<, <,<,=, >, >, >}. The following are equivalent: 


(a) pop. 


(b) There exists a vector u > 0 in R” with Pus wu. 


(c) There exists a vector u > 0 in R” with Pus yu. 
bb 


In particular, 


_ (Px); ; (Px); 
Pe = Max min =min max 
x>0 {i:x; >0} Xj x>0 {i:x; >0} Xj 
_ (Px); ‘ (Px); 
= max min = min max ; 


x>0 1 xX; x>0 1 xX; 
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10. 


11. 


Since p(P?) = p(P), the above properties (and characterizations) of p can be expressed by 
applying the above conditions to P’. 

Consider the sets Q(P) = {u > 0:4x > 0, Px > ux}, Q)(P) = {uw > 0:45x > 0, Px > 
px}, U(P) = {uw > 0:4x > 0, Px < ux}, D)(P) = {u = 0:5x > 0, Px < px}; these sets were 
named the Collatz—Wielandt sets in [BS75], giving credit to ideas used in [Col42], [Wie50]. The 
above properties (and characterizations) of p can be expressed through maximal/minimal elements 
of the Collatz—Wielandt sets of P and P’. (For further details see Chapter 26.) 


. (Bounds on the Spectral Radius) Let A be a complex n x n matrix and let P be an irreducible 


nonnegative m x n matrix such that |A| < P. 
(a) p(A) < p(P). 


(b) [Wie50], [Sch96] p(A) = pe(P) if and only if there exist a complex number p with || = 1 
and a complex diagonal matrix D with |D| = I such that A = D7! P D; in particular, in this 
case |A| = P. 


(c) If Ais real and yz and o € {<, <} satisfy the condition stated in 7b or 7c, then p(A) < wp. 
(d) If A is real and wx and © € {<, =} satisfy the condition stated in 7b or 7c, then p(A) < wp. 


. (Functional Inequalities) Consider the function p(.) mapping irreducible, nonnegative n x n ma- 


trices to their spectral radius. 


(a) p(.) is strictly increasing in each element (of the domain matrices), i.e., if A and B are irre- 
ducible, nonnegative n x n matrices with A > B > 0, then p(A) > p(B). 


(b) [Coh78] p(.) is jointly) convex in the diagonal elements, i.e., if A and D are n x n matrices, 
with D diagonal, A and A + D nonnegative and irreducible and if 0 < a < 1, then p[aA + 
(l—a@)(A+ D)] < ap(A)+ 1 —a@)p(A+ D). 


For further functional inequalities that concern the spectral radius see Fact 8 of Section 9.3. 
(Taylor Expansion of the Perron Value) [HRR92] The function p(.) mapping irreducible nonnegative 
n X nmatrices X = [x;;] to their spectral radius is differentiable of all orders and has a converging 
Taylor expansion. In particular, if P is an irreducible nonnegative n x n matrix with spectral radius 
p and corresponding positive right and left eigenvectors v = [v;] and w = [wj], normalized so 
that wv = 1, and if F is ann x n matrix with P + €F > 0 for all sufficiently small positive €, 
then p(P + €F) = oy pre* with po = p, 1 = WFV, p2 = WF (pI — P)* Fv, p3 = wF (pI — 
P)*(wFvI — F)(pI — P)* Fv; in particular, OI yp = WiVj. 

An algorithm that iteratively generates all coefficients of the above Taylor expansion is available; 
see [HRR92]. 
(Bounds on the Ergodicity Coefficient) [RT85] Let P be an irreducible nonnegative n x n matrix 
with spectral radius p, corresponding positive right eigenvector v, and ergodicity coefficient T; 
let D be a diagonal n x n matrix with positive diagonal elements; and let ||.|| be a norm on R”. 
Then 

t< max Ix’ D“' PD]. 


xeR’ ||x||<1,x7 D-'v=0 


Examples: 


1. 


We illustrate Fact 1 using the matrix 


BIO BIR Ni 
Cole BIR AIR 
Cole NI Wl 


> 48 > 48 
w = [57, 18, 32] are positive right and left eigenvectors, respectively, corresponding to eigenvalue 


The eigenvalues of P are 1 x ( 3 v3) 5 ( 3+ /33),80 p(A) = 1.Also,v = [1, 1,1] and 
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land 


The spectral radius of P is 1 with corresponding right and left eigenvectors v = (1,1)? and 
= (1,1), respectively, the period of P is 2, and (I — P)? = ff. Evidently, 


be I ifmiseven 
~ |) P- ifmisodd. 


In particular, 


lim P?"=I and lim P?"t!= Pp 
m—>0o m—> oo 


and 


1 I+P sor 3D 
se poy P| | =voetw for each m= 0,1,..., 


assuring that, trivially, 


: 15S op 55 4 
pea ea 


In this example, t(P) is 0 (as the maximum over the empty set) and the convergence of the above 
sequences is geometric with rate 0. Finally, 


eo phe ee if m is even 
7 malt P) + Af if m is odd, 
implying that 
lim P™ = Vee) = v(wv)'w (C,1) 
moo 2 
and 


1 
I+P I-—P 

: : = 

jim. Le m( 5 Ve 7 (C,l). 


1 
0 1 . Then p(P+¢€F) = 


—1)k4+1 ve! 
Vlt+te=l+ se of yo ree aa a 


. [RT85, Theorem 4.1] and [Hof67] With ||.|| as the 1-norm on R" and d,,...,d, as the (positive) 


diagonal elements of D, the bound in Fact 11 on the coefficient of ergodicity t(P) of P becomes 


dy|v; P, 7 P, : 
1! dareaw, (% KIVs Prk — v ) 
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With D = I, a relaxation of this bound on t(P) yields the expression 


P, 


n n 
; ‘ if Vj PijVj 
<min 4 p— > min( ~~ ),S° max ( +} - 
j=l 1 Vi qa L Vv 


5. [RT85, Theorem 4.3] For a positive vector u € IR”, consider the function M":IR” — R defined 
for a € R” by 


M*(a) = max{x!a:x € R”, |x|] < 1,x’u = 0}. 


This function has a simple explicit representation obtained by sorting the ratios “‘, i.e., identifying 
uy 


a permutation j(1),..., j7(#) of 1,..., such that 
a; a; a; 
j(1) < j(2) Bene i) 
Uj(1) Uj(2) U j(n) 


With k* as the smallest integer in {1,...,} such that 2 ae Uj(p) > dopey Ue and 


n k* 
wM=1+ Sou - 25° uj0p) ; 
t=l p=l 


we have that 


k*-1 n 
M*(a) = S 7 aj(p) + Haj) — D> 4j0p)- 
p=1 p=kt+1 


With ||.|| as the oo-norm on R" and (D~!PD),,...,(D~'PD),, as the columns of D~! P D, the 
bound in Fact 11 on the coefficient of ergodicity t(P) of P becomes 


max M?'“((D~!PD),]. 


r=1 


9.3. Reducible Matrices 


Definitions: 


For a nonnegative n x n matrix P with spectral radius p: 

A basic class of P is an access equivalence class B of P with p(P[B]) = p. 

The period of an access equivalence class C of P (also known as the index of imprimitivity of C) is 
the period of the (irreducible) matrix P[C]. 

The period of P (also known as the index of imprimitivity of P ) is the least common multiple of the 
periods of its basic classes. 

P is aperiodic if its period is 1. 

The index of P, denoted vp, is vp(p). 

The co-index of P, denoted ip, is max{vp(A):A € o(P),|A| = panda ¥ pe} (with the maximum 
over the empty set defined as 0). 

The basic reduced digraph of P, denoted R*(P), is the digraph whose vertex-set is the set of basic 
classes of P and whose arcs are the pairs (B, B’) of distinct basic classes of P for which there exists a simple 
walk in R[I'(P)] from B to B’. 

The height of a basic class is the largest number of vertices on a simple walk in R*(P) which ends at B. 

The principal submatrix of P at a distinguished eigenvalue A, denoted P [A], is the principal submatrix 
of P corresponding to a set of vertices of I(P) having no access to a vertex of an access equivalence class 
C that satisfies o(P[C]) > A. 
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P is convergent or transient if lim,,... P” = 0. 


P is semiconvergent if lim,,,o. P'™ exists. 


P is weakly expanding if Pu > u for some u > 0. 


P is expanding if for some Pu > u for some u > 0. 


Ann Xn matrix polynomial of degree d in the (integer) variable m is a polynomial in m with coefficients 
that are n x n matrices (expressible as S(m) = par m' B, with B,,..., Bg asn x n matrices and By # 0). 


Facts: 


Facts requiring proofs for which no specific reference is given can be found in [BP94, Chap. 2]. 


1. The set of basic classes of a nonnegative matrix is always nonempty. 
2. (Spectral Properties of the Perron Value) Let P be a nonnegative n x n matrix with spectral radius 
p and index v. 


(a) 
(b) 


(g) 


Frol12] p is an eigenvalue of P. 


Fro12] There exist semipositive right and left eigenvectors of P corresponding to 9, i.e., p is 
a distinguished eigenvalue of both P and P?. 


Rot75] v is the largest number of vertices on a simple walk in R*(P). 


Rot75] For each basic class B having height h, there exists a generalized eigenvector v® in 
N?(P), with (v®); > Oifi H Band (v*); = 0 otherwise. 

Rot75] The dimension of Ni(P) is the number of basic classes of P. Further, if B),..., Bp 
are the basic classes of P and v"',...,v: are generalized eigenvectors of P at p that satisfy the 
conclusions of Fact 2(d) with respect to B,,..., B,, respectively, then v*1,...,v3? forma basis 
of Nj (P). 


[RiSc78, Sch86] If B,,..., Bp is an enumeration of the basic classes of P with nondecreasing 
heights (in particular, s < t assures that we do not have B; +> B,), then there exist generalized 
eigenvectors v5,...,v> of P at p that satisfy the assumptions and conclusions of Fact 2(e) 
and a nonnegative p x p upper triangular matrix M with all diagonal elements equal to p, 


such that 
P[v3,...,v?>] = [v?!,...,v?°]M 


(in particular, v®',...,v?» is a basis of N; (P)). Relationships between the matrix M and the 
Jordan Canonical Form of P are beyond the scope of the current review; see [Sch56], [Sch86], 
[HS89], [HS91a], [HS91b], [HRS89], and [NS94]. 


[Vic85], [Sch86], [Tam04] If By,..., B, arethe basicclasses of P havingheight 1 andv™,...,v® 
are generalized eigenvectors of P at p that satisfy the conclusions of Fact 2(d) with respect to 
By,..., B,, respectively, then v*i,...,v% are linearly independent, nonnegative eigenvectors 
of P at p that span the cone (R{)"N N, (P); that is, each vector in the cone (R{)"N N) (P)isa 
linear combination with nonnegative coefficients of v"', ..., v8 (in fact, the sets {av®: :a~ > 0} 


fors = 1,...,r are the the extreme rays of the cone (R{)" ‘al Nie 


3. (Spectral Properties of Eigenvalues A # p(P) with |A| = p(P)) Let P bea nonnegative n x n matrix 
with spectral radius p, index v, co-index *, period q, and coefficient of ergodicity tT. 


(a) 


[Rot81a] The following are equivalent: 
i. (A €o(P) \ {pe}: |A| = p} = 9. 
ii, D=O. 


iii. P is aperiodic (q = 1). 
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(b) [Rot8la] If A € o(P) \ {p} and |A| = p, then (Gy = 1 for someh € {2,...,n}; further, 
q = minfh = Detesnels ye = 1foreacha € o(P) \ {eo} with |A| = op} < n (here the 
minimum over the empty set is taken to be 1). 


(c) [Rot80] IfA € o(P)\{p}and |A| = p, then vp (A) is bounded by the largest number of vertices 
ona simple walk in R*(P) with each vertex corresponding to a (basic) access equivalence class 
C that has 4 € o( P[C]); in particular, 5 < v. 


4. (Distinguished Eigenvalues) Let P be a nonnegative n x n matrix. 


(a) [Vic85] A isa distinguished eigenvalue of P ifand only if there isa final set C with p(P[C]) = A. 


It is noted that the set of distinguished eigenvalues of P and P? need not coincide (and the 
above characterization of distinguished eigenvalues is not invariant of the application of the 
transpose operator). (See Example 1 below.) 


(b) [HS88b] If A is a distinguished eigenvalue, vp (A) is the largest number of vertices on a simple 
walk in R*(P[A]). 


(c) [HS88b] If ~ > 0, then w < min{A:A isa distinguished eigenvalue of P} if and only if there 
exists a vector u > 0 with Pu > yu. 


(For additional characterizations of the minimal distinguished eigenvalue, see the concluding 
remarks of Facts 12(h) and 12(i).) 


Additional properties of distinguished eigenvalues 2 of P that depend on P[A] can be found in 
[HS88b] and [Tam04]. 

5. (Convergence Properties of Powers) Let P be anonnegative n x n matrix with positive spectral radius 
p, index v, co-index *, period q, and coefficient of ergodicity t (for the case where p = 0, see Fact 
12(j) below). 


(a) [Rot81a] There exists an n x n matrix polynomial S(m) of degree v — 1 in the (integer) variable 
m such that limy,-. 90 ee — S(m)] = 0(C, p) for every p > 0; further, if P is aperiodic, this 
limit holds as a regular limit and the convergence is geometric with rate 5 < 1. 


(b) [Rot81la] There exist matrix polynomials S°(1),...,S4~!(m) of degree v — 1 in the (integer) 
variable m, such that for each k = 0,...,q — 1, limnsool(Z"™4t* — S'(m)] = 0 and the 
convergence of these sequences to their limit is geometric with rate (7 )4 < 1. 


(c) [Rot81a] There exists a matrix polynomial T(m) of degree v in the (integer) variable m with 
limn—>oo Soe — T(m)] = 0 (C, p) for every p > 5; further, if P is aperiodic, this limit 
holds as a regular limit and the convergence is geometric with rate ; <i. 

(d) [FrSc80] The limit of at [I+ 7 feet gall exists and is semipositive. 

(e) [Rot81b] Let x = [x;] be a nonnegative vector in R" and let i € (nm). With K(i,x) = {j € 
(n):j i} A {7 € (n): ut > j for some u € (n) with x, > 0}, 

r(ilx, P) =inffa > 0: lim a~"(P™x); = 0} = p(P[K(i,x)]) 


m—> oo 
and ifr = r(i|x, P) > 0, 


k(i|x, P) = inf{k =0,1,...: lim, m *r—™(P™x); = 0} = vetxiy) (1). 
Explicit expressions for the polynomials mentioned in Facts 5(a) to 5(d) in terms of characteristics 
of the underlying matrix P are available in Fact 12(a)ii for the case where v = 1 and in [Rot81a] 
for the general case. In fact, [Rot81a] provides (explicit) polynomial approximations of additional 
high-order partial sums of normalized powers of nonnegative matrices. 
6. (Bounds on the Perron Value) Let P be a nonnegative n x n matrix with spectral radius p and let jz 
be a nonnegative scalar. 
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(a) Foro € {<,<,=,2,>}, 
[Pu wu for some vector u > 0) > [pop]; 


further, the inverse implication holds for o as <, implying that 


_ (Ax) 
p =max min : 
x20 {i:xj>0} Xx; 


(b) Foro € {<,<,=,>,>}, 


abs pat) 


[po uw] => [Puc pu for some vector u > 0]; 


further, the inverse implication holds for ¢ as > . 
(c) p < wifand only if Pu < pu for some vector u > 0. 


Since p(P?) = p(P), the above properties (and characterizations) of p can be expressed by 
applying the above conditions to P?. (See Example 3 below.) 

Some of the above results can be expressed in terms of the Collatz—Wielandt sets. (See Fact 7 of 
Section 9.2 and Chapter 26.) 


. (Bounds on the Spectral Radius) Let P be a nonnegative n x n matrix and let A be a complex n x n 


matrix such that |A| < P. Then p(A) < e(P). 


. (Functional Inequalities) Consider the function o(.) mapping nonnegative n x n matrices to their 


spectral radius. 


(a) p(.) is nondecreasing in each element (of the domain matrices); that is, if A and B are non- 
negative n x n matrices with A > B > 0, then p(A) > p(B). 


(b) [Coh78] p(.) is jointly) convex in the diagonal elements; that is, if A and D are n x n matrices, 
with D diagonal, Aand A+ D nonnegative, andif0 <a < 1,thenp[wA+(1—a@)(A+D)] < 
ap(A)+(1—a)p(A+ D). 

(c) [EJD88] If A = [a;;] and B = [b;;] are nonnegative n x n matrices,0 < a < landC = [c;j] 
with cj; = ax bi," for each i, j = 1,...,n, then p(C) < p(A)%p(B)!™. 

Further functional inequalities about p(.) can be found in [EJD88] and [EHP90]. 


. (Resolvent Expansions) Let P be a nonnegative square matrix with spectral radius p and let u > p. 


Then wI — P is invertible and 


(ul-P)t=>> 2 oS 0 
lL 


(the invertibility of I — P and the power series expansion of its inverse do not require nonnegativity 
of P). 

For explicit expansions of the resolvent about the spectral radius, that is, for explicit power 
series representations of [(z + p)I — P]~! with |z| positive and sufficiently small, see [Rot81c], 
and [HNR90] (the latter uses such expansions to prove Perron—Frobenius-type spectral results for 
nonnegative matrices). 

(Puiseux Expansions of the Perron Value) [ERS95] The function p(.) mapping irreducible non- 
negative n x n matrices X = [x;j] to their spectral radius has a converging Puiseux (fractional 
power series) expansion at each point; i.e., if P is a nonnegative n x n matrix and if F isann x n 
matrix with P + €F > 0 for all sufficiently small positive €, then p(P + €F) has a representation 
bea pe*/4 with pp = p(P) and q as a positive integer. 

(Bounds on the Ergodicity Coefficient) [RT85, extension of Theorem 3.1] Let P be a nonnegative 
n X n matrix with spectral radius p, corresponding semipositive right eigenvector v, and ergodicity 
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coefficient t, let D be a diagonal n x n matrix with positive diagonal elements, and let ||.|| be a 
norm on R”. Then 


t< max Ix’ D“! PD]. 


xelR’ ||x||<1,x? D-'v=0 


12. (Special Cases) Let P be a nonnegative n x n matrix with spectral radius p, index v, and period q. 


(a) 


Sc 
ce 


(d) 


(e) 


(h) 


(Index 1) Suppose v = 1. 
i. pI — P hasa group inverse. 


ii. [Rot81a] With P* = I — (pI — P)(pI — P)*, all of the convergence properties stated in 
Fact 6 of Section 9.2 apply. 
iii. If o > 0, then a is bounded in m (element-wise). 


iv. p = Oifand only if P = 0. 


(Positive eigenvector) The following are equivalent: 
i. P has a positive right eigenvector corresponding to p. 
ii. The final classes of P are precisely its basic classes. 
iii. There is no vector w satisfying w! P < pw’. 
Further, when the above conditions hold: 
i. v = 1 and the conclusions of Fact 12(a) hold. 


ii. If P satisfies the above conditions and P ¥ 0, then p > 0 and there exists a diagonal 
matrix D having positive diagonal elements such that S = a P D is stochastic (that is, 
S > Oand S1 = 1; see Chapter 4). 


[Sch53] There exists a vector x > 0 with Px < px if and only if every basic class of P is final. 
(Positive generalized eigenvector) [Rot75], [Sch86], [HS88a] The following are equivalent: 
i. P hasa positive right generalized eigenvector at p. 
ii. Each final class of P is basic. 
iii. Pu > puforsomeu > 0. 
iv. Every vector w > 0 with w’P < pw! must satisfy w'P = pw’. 
v. ¢ is the only distinguished eigenvalue of P. 
(Convergent/Transient) The following are equivalent: 
i. P is convergent. 
i. p <1. 
iti. I — P is invertible and (I — P)~! > 0. 
iv. There exists a positive vector u € R" with Pu < u. 
Further, when the above conditions hold, (I — P)~! = Sey Pes: 
(Semiconvergent) The following are equivalent: 
i. P is semiconvergent. 
ii. Either p < 1 or op = v = 1 and 1 is the only eigenvalue A of P with |A| = 1. 
(Bounded) P™ is bounded in m (element-wise) if and only if either p < 1 or p = 1 and v=1. 
(Weakly Expanding) [HS88a], [TW89] [DRO5] The following are equivalent: 
i. P is weakly expanding. 
ii. There is no vector w € R" withw > Oandw! P < w?. 


iii. Every distinguished eigenvalue A of P satisfies X > 1. 
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iv. Every final class C of P has p(P[C]) > 1 
v. If C isa final set of P, then p(P[C]) > 1. 


Given jz > 0, the application of the above equivalence to * yields characterizations of instances 
where each distinguished eigenvalue of P is bigger than or equal to jz. 


(i) (Expanding) [HS88a], [TW89] [DRO5] The following are equivalent: 
i. P is expanding. 
ii. There exists a vector u € R” with u > Oand Pu > u. 
iii. There is no vector w € R" with w > Oandw!P <w’. 
iv. Every distinguished eigenvalue A of P satisfies A > 1. 
v. Every final class C of P has p(P[C]) > 1. 
vi. If C isa final set of P, then p(P[C]) > 1. 


Given jz > 0, the application of the above equivalence to 7 yields characterizations of instances 
where each distinguished eigenvalue of P is bigger than ju. 


(j) (Nilpotent) The following are equivalent conditions: 


i. P is nilpotent; that is, P’” = 0 for some positive integer m. 


ii. P is permutation similar to an upper triangular matrix all of whose diagonal elements are 0. 


ili, p = 0. 
iv. P"=0 
v. P’=0. 


(k) (Symmetric) Suppose P is symmetric. 


u’ Pu 
wu * 


i. P = MaXy>0 
i. p= ae for u > 0 ifand only if u is an eigenvector of P corresponding to p. 


iii. [CHR97, Theorem 1] For u,w > 0 with w; = /u;(Pu); fori = 1,...,n, wpa < wipw 


and equality holds if and only if u[S] is an eigenvector of P[S] corresponding to p, where 
S = {i:u; > 0}. 


Examples: 


1. We illustrate parts of Fact 2 using the matrix 


ar) 

II 
ooooown 
CoCo CoOoON NM 
ooocoronN 
OrRrRNOCOS 
Orroc#$ceoe 
Pr OCC CO 


The eigenvalues of P are 2,1, and 0; so, p(P) = 2 € o(P) as is implied by Fact 2(a). The 
vectors v = [1,0,0,0,0,0]7 and w = [0,0,0, 1, 1,1] are semipositive right and left eigenvectors 
corresponding to the eigenvalue 2; their existence is implied by Fact 2(b). 

The basic classes are Bj = {1}, B} = {2} and B3; = {4,5}. The digraph corresponding to P, its 
reduced digraph, and the basic reduced digraph of P are illustrated in Figure 9.1. From Figure 9.1(c), 
the largest number of vertices in a simple walk in the basic reduced digraph of P is 2 (going from B; 
to either By or B3); hence, Fact 2(c) implies that vp (2) = 2. The height of basic class B, is 1 and the 
height of basic classes B and B; is 2. Semipositive generalized eigenvectors of P at (the eigenvalue) 
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tI} 


{2} 


{4,5} 


(a) (b) (c) 


FIGURE9.1 (a) The digraph I'(P), (b) reduced digraph R[I'(P)], and (c) basic reduced digraph R*(P). 


2 that satisfy the assumptions of Fact 2(f) are wu?! = [1,0,0,0,0,0]7, u® = [1,1,0,0,0,0]", and 
u® = [1,0,2, 1, 1,0]7. The implied equality 


Plu®,...,u??] = [u?',...,u??]M 

of Fact 2(f) holds as 

2 2 2 0 0 0 1 1 ii 2 4 6 1 1 ii 

0 2 0 0 0 0; }0 1 0 0 2 0 0 1 0 724 

0 01 2 0 0); }0 0 2 = 0 0 4 - 0 0 2 020 

00 0 41 41 01/0 0 1! +J0 0 2! Jo 0 1 002 : 

0001 1 i441 0 0 1 0 0 2 0 0 1 

0 00 0 0 41 0 0 0 0 0 O 0 0 0 
In particular, Fact 2(e) implies that u®',u®?,u®? form a basis of Ne = N3. We note that 
while there is only a single basic class of height 1, dim[N}(P)] = 2 and w3!,2u® —uB = 


[—1,2, —2, -1,—1,0]7 form a basis of Ni(P). Still, Fact 2(g) assures that (IRZ)" Ni (P) is the 
cone {au?! : @ > 0} (consisting of its single ray). 

Fact 4(a) and Figure 9.1 imply that the distinguished eigenvalues of P are 1 and 2, while 2 is the 
only distinguished eigenvalue of P?. 


0 1 ; : ; F 
2. Let H = 1 ol: properties of H were demonstrated in Example 2 of section 9.2. We will demon- 


strate Facts 2(c), 5(b), and 5(a) on the matrix 


The spectral radius of P is 1 and its basic classes of P are B,; = {1,2} and B, = {3,4} with B, 
having access to B. Thus, the index of 1 with respect to P, as the largest number of vertices on a 
walk of the marked reduced graph of P, is 2 (Fact 2(c)). Also, as the period of each of the two basic 
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classes of P is 2, the period of P is 2. To verify the convergence properties of P, note that 


a ee if m is even 
0 I 
Pp" = 
H mi é 5 
| al if m is odd, 


immediately providing matrix—polynomials $°(m) and S!(m) of degree 1 such that limj,. 5. P?" — 
S°(m) = 0 and limo P?”"*t! — S'(m) = 0. In this example, t(P) is 0 (as the maximum over 
the empty set) and the convergence of the above sequences is geometric with rate 0. 

The above representation of P™ shows that 


H™ mH™t! 
PpPv= 


and Example 2 of section 9.2 shows that 


I+H 
TN ee a 3 (C1). 


m—> 00 2 20 4 


We next consider the upper-right blocks of P”. We observe that 


LS pip 5 mi 4 (m- 2H if m is even 
>. »D2I= 
i (m= 4 PDE if m is odd, 
eee if m is even 
~) m(I+H) I I-H ; : 
ioe ep ae if m is odd, 
implying that 
es I+H I+H 
lim — P'[B,,B m =0 (Gl). 
Ji dy PB Bl (=) +4 -0 cn 
Asm—1= 45°") t for each m = 1,2,..., the above shows that 
ee or ace I+H 
jim — > {P'tB Bal -1 (22) } = 0 on, 


t=0 


and, therefore (recalling that (C,1)-convergence implies (C,2)-convergence), 


& 5 -=25m 25m 
5 5 —.25m —.25m 
Hi. 7 
me ey 00 5 5 O(C2) 
0 @ 5 5 


. Fact 6 implies many equivalencies, in particular, as the spectral radius of a matrix equals that of its 


transpose. For example, for a nonnegative n x m matrix P with spectral radius p and nonnegative 
scalar jy, the following are equivalent: 


(a) p <p. 
(b) Pu < pu for some vector u > 0. 


(c) w! P < yw! for some vector w > 0. 
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(d) Pu < pu for some vector u > 0. 
(e) w’ P < pw’ for some vector w > 0. 
(f) There is no vector u > 0 satisfying Pu > ju. 


(g) There is no vector w > 0 satisfying w! P > ww’. 


9.4 Stochastic and Substochastic Matrices 


(For more information about stochastic matrices see Chapter 54 (including examples).) 


Definitions: 


Asquare n x n matrix P = [ p;;] is stochastic if it is nonnegative and P1 = 1 where1 = [1,..., 1]7 eR". 
(Stochastic matrices are sometimes referred to as row-stochastic, while column-stochastic matrices are 
matrices whose transpose is (row- stochastic.) 

A square n x n matrix P is doubly stochastic if both P and its transpose are stochastic. The set of 
doubly stochastic matrices of order n is denoted Qy. 

A square n x n matrix P is substochastic if it is nonnegative and P1 < 1. 

A transient substochastic matrix is also called stopping. 

An ergodic class of a stochastic matrix P is a basic class of P. 

A transient class of a stochastic matrix P is an access equivalence class of P which is not ergodic. 

A state of an n x n stochastic matrix P is an indexi € {1,...,n}. Such a state is ergodic or transient 
depending on whether it belongs to an ergodic class or to a transient class. 

A stationary distribution of a stochastic matrix P is a nonnegative vector 7 that satisfies 771 = 1 and 


ni P=, 


Facts: 


Facts requiring proofs for which no specific reference is given follow directly from facts in Sections 9.2 and 
9.3 and/or can be found in [BP94, Chap. 8]. 


1. Let P = [p;j] be ann x n stochastic matrix. 


(a) p(P) = 1,1 € R" isaright eigenvector of P corresponding to 1 and the stationary distributions 
of P are nonnegative eigenvectors of P corresponding to 1. 


(b) vp) =1. 
(c) I — P hasa group inverse. 
(d) The height of every ergodic class is 1. 


(e) The final classes of P are precisely its ergodic classes. 


i. For every ergodic class C, P has a unique stationary distribution 7° of P with (4°); > 0 
ifi € C and (x©); = 0 otherwise. 

ii. IfC',...,C? are the ergodic classes of P, then the corresponding stationary distributions 
ne... ae (according to Fact 1(f)i above) form a basis of the set of left eigenvectors 
of P corresponding to the eigenvalue 1; further, every stationary distribution of P is a 


convex combination of these vectors. 
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i. Let T and R be the sets of transient and ergodic states of P, respectively. The matrix 
I — P[T] is nonsingular and for each ergodic class C of P, the vector u© given by 


e ifK =C 
(u°)[K] = 40 ifK =R\C 
(I—P[T])"P[T,Cle ifK=T 


is a right eigenvector of P corresponding to the eigenvalue 1; in particular, (u°); > 0 if i 
has access to C and (u©); = 0 ifi does not have access to C. 


ii, If C!,...,C? are the ergodic classes of P, then the corresponding vectors u, ul 


(referred to in Fact 1(g)i above) form a basis of the set of right eigenvectors of P corre- 
sponding to the eigenvalue 1; further, S7?_, u\' = e. 


Let C!,...,C? be the ergodic classes of P, nr, ..., 7° the corresponding stationary dis- 
tributions (referred to in Fact 1(f)i above), and u’',...,u°” the corresponding eigenvectors 
referred to in Fact 1(g)i above. Then the matrix 


1 Pp 
Ppt= [uc an | 


TT 


is stochastic and satisfies P*[(n),C] = O if C is a transient class of P, P*[i,C] 4 Oif C is an 
ergodic class and i has access to C, and P*[i,C] = 0 if C is an ergodic class and i does not 
have access to C. 


The matrix P* from Fact 1(h) above has the representation I — (I — P)*(I — P); further, 
I — P + P* is nonsingular and (I — P)* = (I — P + P*)~'(I — P*). 


With P* as the matrix from Fact 1(h) above, lim,,.., P™ = P* (C,1); further, when P 
is aperiodic, this limit holds as a regular limit and the convergence is geometric with rate 
t(P) <1. 


With P* as the matrix from Fact 1(h) above, limjy_. 06 pray Pt — mP* = (I — P)* (C1); 
further, when P is aperiodic, this limit holds as a regular limit and the convergence is geometric 
with rate t(P) < 1. 


With D a diagonal n x n matrix with positive diagonal elements and ||.|| a norm on R", 


t(P) < max Ix’ D“! PD]. 
xelR” |[x\|<1,x7 D-11=0 


In particular, with ||.|| as the 1-norm on R” and D = I, the above bound specializes to 


t(P)< max Pas Ball min | 1 Son prs max pa — 1} 


r,s=1,....n,rAs 
ie fe 2 k=l k=1 


(cf. Fact 11 of section 9.3 and Example 4 of section 9.2). 


For every 0 < a < 1, Py = (1—a@)I + a@P is an aperiodic stochastic matrix whose ergodic 
classes, transient classes, stationary distributions, and the vectors of Fact 1(g)i coincide with 
those of P. In particular, with P* and P* as the matrices from Fact 1(h) corresponding to P 
and P,, respectively, limy,o0(Pa)” = P% = P*. 
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2. Let P be an irreducible stochastic matrix with coefficient of ergodicity tT. 
(a) P has a unique stationary distribution, say zr. Also, up to scalar multiple, 1 is a unique right 
eigenvector or P corresponding to the eigenvalue 1. 
(b) With 7 as the unique stationary distribution of P, the matrix P* from Fact 1(h) above equals 


17. 


3. A doubly stochastic matrix is a convex combination of permutation matrices (in fact, the n x n 
permutation matrices are the extreme points of the set Q,, of n x n doubly stochastic matrices). 
4. Let P beann x n substochastic matrix. 
(a) p(P) <1. 
(b) vp(1) <1. 
(c) I — P hasa group inverse. 
(d) The matrix P* = I — (I — P)*(I — P) is substochastic; further, J — P + P* is nonsingular 
and (I — P)* = (IJ — P+ P*)7'(I — P*). 
(e) With P* as in Fact 4(d), lim... P™ = P* (C,1); further, when every access equivalence class 
C with p(P[C]) = 1 is aperiodic, this limit holds as a regular limit and the convergence is 
geometric with rate max{|A|:4 € o(P) and |A| 4 1} < 1. 
(f) With P* as the matrix from Fact 4(d) above, limyy_. oo ar Pt — mP* = (I — P/)* (C1); 
further, when every access equivalence class C with p(P[C]) = 1is aperiodic, this limit holds as 


a regular limit and the convergence is geometric with rate max{|A|:A € o(P)and|A| 4 1} < 1. 
(g) The following are equivalent: 
i. P is stopping. 
ii, p(P) <1. 
iii. I — P is invertible. 
iv. There exists a positive vector u € R" with Pu < u. 


Further, when the above conditions hold, (I — P)~! = yee 0. 


9.5 M-Matrices 


Definitions: 


Ann x nreal matrix A = [a;;] is a Z-matrix if its off-diagonal elements are nonpositive, i.e., if aj; < 0 
for alli, 7 =1,...,n withi # j. 

An Mo-matrix is a Z-matrix A that can be written as A = sI — P with P as a nonnegative matrix and 
with s as a scalar satisfying s > p(P). 

An M-matrix A is a Z-matrix A that can be written as A = sI — P with P as a nonnegative matrix 
and with s asa scalar satisfying s > p(P). 

A square real matrix A is an inverse M-matrix if it is nonsingular and its inverse is an M-matrix. 

A square real matrix A is inverse-nonnegative if it is nonsingular and A~' > 0 (the property is 
sometimes referred to as inverse-positivity). 

A square real matrix A has a convergent regular splitting if A has a representation A = M — N such 
that N > 0, M invertible with M~' > 0 and M™'N is convergent. 

A square complex matrix A is positive stable if the real part of each eigenvalue of A is positive; A is 
nonnegative stable if the real part of each eigenvalue of A is nonnegative. 

An n x n complex matrix A = [a;;] is strictly diagonally dominant (diagonally dominant) if 
laii| > SO jaaja laiil Vail = ha1,j¢i lai) fori = L...50. 

Ann x n M-matrix A satisfies property C if there exists a representation of A of the form A = sI — P 
with s > 0, P > 0 and semiconvergent. 
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Facts: 
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Facts requiring proofs for which no specific reference is given follow directly from results about nonnegative 
matrices stated in Sections 9.2 and 9.3 and/or can be found in [BP94, Chap. 6]. 


1. Let Abe ann x n real matrix with n > 2. The following are equivalent: 


(a) 


(b) 
(c) 
(d) 
(e) 
(f) 


A is an M-matrix; that is, A is a Z-matrix that can be written as sI — P with P nonnegative 
ands > e(P). 


A is a nonsingular Mo-matrix. 

For each nonnegative diagonal matrix D, A + D is inverse-nonnegative. 
For each uw > 0, A+ pI is inverse-nonnegative. 

Each principal submatrix of A is inverse-nonnegative. 


Each principal submatrix of A of orders 1, ... , is inverse-nonnegative. 


2. Let A = [a;;] be ann x n Z-matrix. The following are equivalent:* 


(a) 


A is an M-matrix. 

Every real eigenvalue of A is positive. 

A+ Dis nonsingular for each nonnegative diagonal matrix D. 

All of the principal minors of A are positive. 

For each k = 1,...,n, the sum of all the k x k principal minors of A is positive. 


There exist lower and upper triangular matrices L and U, respectively, with positive diagonal 
elements such that A = LU. 


A is permutation similar to a matrix satisfying condition 2(f). 
A is positive stable. 


There exists a diagonal matrix D with positive diagonal elements such that AD + DA? is 
positive definite. 


There exists a vector x > 0 with Ax > 0. 
There exists a vector x > 0 with Ax > 0 and ia ajjxj > Ofori = 1,...,n. 
A is permutation similar to a matrix satisfying condition 2(k). 


There exists a vector x > 0 such that Ax > 0 and the matrix A= (a; j] defined by 


as 1 ifeither a;; 4 0 or (Ax); 4 0 
4 ~~ 10 otherwise 


is irreducible. 


All the diagonal elements of A are positive and there exists a diagonal matrix D such that AD 
is strictly diagonally dominant. 


A is inverse-nonnegative. 


Every representation of A of the form A = M — N with N > 0 and M inverse-positive must 
have M~!N convergent (i.e., o(M7!N) < 1). 


For each vector y > 0, the set {x > 0: A?x < y} is bounded and A is nonsingular. 


*Each of the 17 conditions that are listed in Fact 2 is a representative of a set of conditions that are known to 
be equivalent for all matrices (not just Z-matrices); see [BP94, Theorem 6.2.3]. For additional characterizations of 
M-matrices, see [FiSc83]. 
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10. 


. Let A be an irreducible n x n Z-matrix with n > 2. The following are equivalent: 


(a) Ais an M-matrix. 
(b) Aisanonsingular and Aq! > 0. 


(c) Ax > 0 for some x > 0. 


. Let A= [a;;] bean n x n M-matrix and let B = [b;;] be ann x n Z-matrix with B > A. Then: 


(a) B isan M-matrix. 
(b) detB > detA. 

(c) AT} > Bo, 

(d) detA < ayy... dyn. 


. If P isan inverse M-matrix, then P > 0 and I(P) is transitive; that is, if (v, w) and (u, w) are arcs 


of I'(P), then so is (v, w). 


. Let A be ann x n real matrix with n > 2. The following are equivalent: 


(a) Ais a nonsingular Mo-matrix. 
(b) For each diagonal matrix D with positive diagonal elements, A + D is inverse-nonnegative. 


(c) For each w > 0, A+ pI is inverse-nonnegative. 


. Let Abe ann x n Z-matrix. The following are equivalent:* 


(a) Ais an Mo-matrix. 

(b) Every real eigenvalue of A is nonnegative. 

(c) A+ D is nonsingular for each diagonal matrix D having positive diagonal elements. 
(d) For each k = 1,...,n, the sum ofall the k x k principal minors of A is nonnegative. 


(e) A is permutation similar to a matrix having a representation LU with L and U as lower and 
upper triangular matrices having positive diagonal elements. 


(f) A is nonnegative stable. 
(g) There exists a nonnegative matrix Y satisfying Y A‘t! = A* for some k > 1. 


(h) A has a representation of the form A = M — N with M inverse-nonnegative, N > 0 and 
B = M' Nsatisfying N@prange(B*) = N%.prange(A*) and p(B) < 1. 


(i) A has a representation of the form A = M — N with M inverse-nonnegative, M7!N > Oand 
B = M"'!N satisfying Ne prange( B*) = NZ range(A*) and p(B) <1. 


. Let A be an Mo-matrix. 


(a) A satisfies property C if and only if v4(0) < 1. 


(b) A is permutation similar to a matrix having a representation LU with L as a lower triangular 
M-matrix and U as an upper triangular Mo matrix. 


. [BP94, Theorem 8.4.2] If P is substochastic (see Section 9.4), then I — P isan M-matrix satisfying 


property C. 
Let A be an irreducible n x n singular Mo-matrix. 


(a) Ahas rank n — 1. 


(b) There exists a vector x > 0 such that Ax = 0. 


*Each of the 9 conditions that are listed in Fact 7 is a representative of a set of conditions that are known to 
be equivalent for all matrices (not just Z-matrices); see [BP94, Theorem 6.4.6]. For additional characterizations of 
M-matrices, see [FiSc83]. 
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(c) A has property C. 
(d) Each principal submatrix of A other than A itself is an M-matrix. 
(e) [Ax > 0] > [Ax = 0]. 


9.6 Scaling of Nonnegative Matrices 


A scaling of a (usually nonnegative) matrix is the outcome of its pre- and post-multiplication by diagonal 
matrices having positive diagonal elements. Scaling problems concern the search for scalings of given 
matrices such that specified properties are satisfied. Such problems are characterized by: 


(a) The class of matrices to be scaled. 
(b) Restrictions on the pre- and post-multiplying diagonal matrices to be used. 
(c) The target property. 


Classes of matrices under (a) may refer to arbitrary rectangular matrices, square matrices, symmetric 
matrices, positive semidefinite matrices, etc. For possible properties of pre- and post-multiplying diagonal 
matrices under (b) see the following Definition subsection. Finally, examples for target properties under 
(c) include: 


i. The specification of the row- and/or column-sums; for example, being stochastic or being doubly 
stochastic. See the following Facts subsection. 

ii. The specification of the row- and/or column-maxima. 

iii. (For asquare matrix) being line-symmetric, that is, having each row-sum equal to the corresponding 
column-sum. 

iv. Being optimal within a prescribed class of scalings under some objective function. One example 
of such optimality is to minimize the maximal element of a scalings of the form X AX. Also, in 
numerical analysis, preconditioning a matrix may involve its replacement with a scaling that has a 
low ratio of largest to smallest element; so, a potential target property is to be a minimizer of this 
ratio among all scalings of the underlying matrix. 


Typical questions that are considered when addressing scaling problems include: 


(a) Characterizing existence of a scaling that satisfies the target property (precisely of approximately). 

(b) Computing a scaling ofa given matrix that satisfies the target property (precisely or approximately) 
or verifying that none exists. 

(c) Determining complexity bounds for corresponding computation. 


Early references that address scaling problems include [Kru37], which describes a heuristic for finding 
a doubly stochastic scaling of a positive square matrix, and Sinkhorn’s [Sin64] pioneering paper, which 
provides a formal analysis of that problem. The subject has been intensively studied and an aspiration 
to provide a comprehensive survey of the rich literature is beyond the scope of the current review; con- 
sequently, we address only scaling problems where the target is to achieve, precisely or approximately, 
prescribed row- and column-sums. 


Definitions: 


Let A = [a;;] bean m x n matrix. 

A scaling (sometimes referred to as an equivalence-scaling or a DAE -scaling) of A is any matrix of 
the form DAE where D and E are square diagonal matrices having positive diagonal elements; such a 
scaling is a row-scaling of A if E = I and it is a normalized-scaling if det(D) = det(E) = 1. 

Ifm =n, ascaling DAE of A isa similarity-scaling (sometimes referred to asa DAD™" scaling) of A 
if E = D~',and DAE isa symmetric-scaling (sometimes referred to as a DAD scaling) of A if E = D. 
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The support (or sparsity pattern) of A, denoted Struct( A), is the set ofindicesij with a;; 4 0; naturally, 
this definition applies to vectors. 


Facts: 


1. (Prescribed-Line-Sum Scalings) [RoSc89] Let A = [a;;] € R”*” be a nonnegative matrix, and let 
r= [r;] € R” andc = [c;] € R” be positive vectors. 


(a) The following are equivalent: 
i. There exists a scaling B of A with Bl =rand17B =c’. 
ii. There exists nonnegative m x n matrix B having the same support as A with B1 = r and 
17B=c!. 
iii. For every I C {1,...,m}and J C {1,...,m} for which A[I‘, J] = 0, 
ADS 
iel joJ 
and equality holds if and only if A[I, J‘] = 0. 


iv. 17r = 1’ rand the following (geometric) optimization problem has an optimal solution: 
minx! Ay 


subject to: x = [xj] € R”,y = [yj] € R” 


A standard algorithm for approximating a scaling of a matrix to one that has prescribed row- 
and column-sums (when one exists) is to iteratively scale rows and columns separately so as to 
achieve corresponding line-sums. 


(b) Suppose 17r = 17r and x = [%;] and y = [y;] form an optimal solution of the optimization 
problem of Fact 1(d). Let A = ses and let X ¢ R™* and Y € R"*" be the diagonal matrices 
having diagonal elements X;; = 3! and Y;; = yj. Then B = X AY isa scaling of A satisfying 
Bl=rand1’B=c’!. 


(c) Suppose X € R™*™ and Y € R"*" are diagonal matrices such that B = X AY isa scaling of A 
satisfying B1 = rand 17 B = c!. Then 1’r = 1’ rand with 


and 


the vectors x = [x;] € R” andy = [y;] € R” with x; = AX; fori = 1,.. .,mand jj = Yj; 
for j = 1,..., are optimal for the optimization problem of Fact 1(d). 


2. (Approximate Prescribed-Line-Sum Scalings) [RoSc89] Let A = [aj;] € R*" be a nonnegative 
matrix, and let r = [r;] € R” andc = [c;] € R" be positive vectors. 


(a) The following are equivalent: 


i. For every € > 0 there exists a scaling B of A with || B1 — rl], < € and ||17B —c"||, <e. 
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ii. There exists nonnegative m xn matrix A’ = [aj ; ] with Struct(A’) CStruct(A) anda; j= 4ij 
for each ij € Struct(A’) such that A’ has a scaling B satisfying B1 = r and VBe=c!. 


iii. For every € > 0 there exists a matrix B having the same support as A and satisfying 
Bl —r|l, < and ||17B —c7]], <e. 


iv. There exists a matrix B satisfying Struct(B) C Struct(A), B1 = rand 1 B=c!. 


v. For every I C {1,...,m}and J C {1,...,m} for which A[I‘, J] = 0, 


Sai aes 


iel G€j 


vi. 1’r = 17 rand the objective of the optimization problem of Fact 2(a)iii is bounded away 
from zero. 


See [NR99] for a reduction of the problem of finding a scaling of A that satisfies ||B1 — rll) < € 
and ||17B —c!||, < € fora given € > 0 to the approximate solution of geometric program that 
is similar to the one in Fact 1(a)iv and for the description of an (ellipsoid) algorithm that solves 
the latter with complexity bound of O(1)(m + n)*In[2 + snl Natori) 
between the largest and smallest positive entries of A. 


], where 8 is the ratio 


9.7 Miscellaneous Topics 


In this subsection, we mention several important topics about nonnegative matrices that are not covered 
in detail in the current section due to size constraint; some relevant material appears in other sections. 


9.7.1 Nonnegative Factorization and Completely Positive Matrices 


A nonnegative factorization of a nonnegative matrix A € R”™*” is a representation A = LR of A with L 
and R as nonnegative matrices. The nonnegative rank of A is the smallest number of columns of L (rows 
of R) in such a factorization. 

A square matrix A is doubly nonnegative if it is nonnegative and positive semidefinite. Such a matrix A 
is completely positive if it has a nonnegative factorization A = BB’; the CP-rank of A is then the smallest 
number of columns of a matrix B in such a factorization. 

Facts about nonnegative factorizations and completely positive matrices can be found in [CR93], 
[BSM03], and [CP05]. 


9.7.2 The Inverse Eigenvalue Problem 


The inverse eigenvalue problem concerns the identification of necessary conditions and sufficient conditions 
for a finite set of complex numbers to be the spectrum of a nonnegative matrix. 
Facts about the inverse eigenvalue problem can be found in [BP94, Sections 4.2 and 11.2] and Chapter 20. 


9.7.3 Nonhomogenous Products of Matrices 


A nonhomogenous product of nonnegative matrices is the finite matrix product of nonnegative matrices 
P, Py... Pm, generalizing powers of matrices where the multiplicands are equal (i.e., P} = P2 = +--+ = Py); 
the study of such products focuses on the case where the multiplicands are taken from a prescribed set. 

Facts about Perron—Frobenius type properties of nonhomogenous products of matrices can be found 
in [Sen81], and [Har02]. 
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9.7.4 Operators Determined by Sets of Nonnegative 
Matrices in Product Form 


A finite set of nonnegative n x n matrices {Ps : 6 € A} is said to be in product form if there exists finite 
sets of row vectors Aj,..., A, such that A = [J/_, A; and for each § = (5),...,5,) € A, Ps is the matrix 
whose rows are, respectively, 5),...,5,. Such a family determines the operators PX and P oe on R" with 
PR**x = maXsea Psx and pring = minsea P;x for each x € R". 

Facts about Perron—Frobenius-type properties of the operators corresponding to families of matrices 


in product form can be found in [Zij82], [Zij84], and [RW82]. 


9.7.5 Max Algebra over Nonnegative Matrices 


Matrix operations under the max algebra are executed with the max operator replacing (real) addition 
and (real) addition replacing (real) multiplication. 

Perron—Frobenius-type results and scaling results are available for nonnegative matrices when consid- 
ered as operators under the max algebra; see [RSS94], [Bap98], [But03], [BS05], and Chapter 25. 
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10.1 Submatrices and Block Matrices 


Definitions: 


Let A € F”*". Then the row indices of A are {1,...,m}, and the column indices of A are {1,...,}. Let 
a, B be nonempty sets of indices with aw C {1,...,m} and B C {1,...,n}. 

A submatrix A[q, 8] is a matrix whose rows have indices a among the row indices of A, and whose 
columns have indices 6 among the column indices of A. A(a, B) = Ala‘, B°], where a is the complement 
of a. 

A principal submatrix is a submatrix A[a, a], denoted more compactly as A[a]. 

Let the set {1,...m} be partitioned into the subsets a,...,@, in the usual sense of partitioning a set 
(so thata; Na; = Y, for alli 4 j,1 < i,j <r, anda, U---Ua, = {1,...,m}), and let {1,...,1} be 
partitioned into the subsets 6),..., B;. 

The matrix A € F”" is said to be partitioned into the submatrices Ala;,6j],1<i<r,1l<j<s. 

A block matrix is a matrix that is partitioned into submatrices A[a;, 8;] with the row indices {1,...,m} 
and column indices {1,...,n} partitioned into subsets sequentially, ie., a; = {1,...,ij},a@2 = {i) + 
1,..., 7}, ete. 

Each entry of a block matrix, which is a submatrix A[qa;, Bj], is called a block, and we will sometimes 
write A = [Ajj] to label the blocks, where Aj = A[qj, Bj]. 

If the block matrix A € F””? is partitioned with ajs and Bjs, 1 <i <r,1< j <s, and the block 
matrix B € F?*" is partitioned with B;s and ys, 1 < j < s,1<k < t, then the partitions of A and B 
are said to be conformal (or sometimes conformable). 


Facts: 
The following facts can be found in [HJ85]. This information is also available in many other standard 
references such as [LT85] or [Mey00]. 


1. Two block matrices A = [Aj] and B = [Bj] in F”*”, which are both partitioned with the same 
ajsand Bjs,1 <i <r,1 <j <s, may be added block-wise, as with the usual matrix addition, so 
that the (i, j) block entry of A+ B is (A+ B)j = Aj + By. 
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2. If the block matrix A € F™*? and the block matrix B € F’*" have conformal partitions, then we 
can think of A and B as having entries, which are blocks, so that we can then multiply A and B 
block-wise to form the m x n block matrix C = AB. Then Cj, = ja Aj Bix, and the matrix C 
will be partitioned with the a;s and ys, 1 <i <r,1<k < t, where A is partitioned with a;s and 
Bjs, 1 <i<r,1 <j <s,and B is partitioned with Bjs and ys, 1< j <s,1<k<t. 

3. With addition and multiplication of block matrices described as in Facts 1 and 2 the usual properties 
of associativity of addition and multiplication of block matrices hold, as does distributivity, and 
commutativity of addition. The additive identity 0 and multiplicative identity I are the same 
under block addition and multiplication, as with the usual matrix addition and multiplication. 
The additive identity 0 has zero matrices as blocks; the multiplicative identity J has multiplicative 
identity submatrices as diagonal blocks and zero matrices as off-diagonal blocks. 

4. Ifthe partitions of A and B are conformal, the partitions of B and A are not necessarily conformal, 
even if BA is defined. 

An An 


(n —k) x (n—k). Then det(A) = (—1)*)* det(Ay,)det(A21). 


5. Let A € F”*" bea block matrix of the form A = , where Aj, is k x k, and Ap is 


Examples: 


Au A12 


1. Let the block matrix A € C"™” given by A = 
Az, Ax 


be Hermitian. Then A,, and A», are 


Hermitian, and Aj; = Aj). 
2. If A = [aj], then A[{z}, {j}] is the 1 x 1 matrix whose entry is aj. The submatrix A({7}, {j}) is the 
submatrix of A obtained by deleting row i and column j of A. 
D2) 5S 
LetA=!-3 0 1 6 1 
2 7 4 5 —7 
2 7 3. -=71- 


3. Then AL(2 8] = fn] = and a2} 0) = |} eee Z 


4. Leta = {1,3} and B = {1, 2,4}. Then the submatrix A[a, 8] = : - } , and the principal 


submatrix A[a] = ; ir 


5. Leta; = {1}, a2 = {2,3} and let 6B; = {1,2}, Bo = {3}, Bs = {4,5}. Then the block matrix, with 
(i, j) block entry Aj = Alq;, Bj],1 <i < 2,1 < j <3,is 


1 | 5 | 3 —l1 
He An Aw Ais) _ | | 
Ax, Ay Ax —3 0 | 1 | 6 1 

2 7 | 4 | 5 —7 

2 —l1 | 0 | 6 —2 
By By By | | : 
. Let B= — . Then th t A 
6. Le | Bao Bs) Boe =" 0 5 3 7 en the matrices 

1 1 | —2 | 2 6 


(of this example) and B are partitioned with the same q;s and fs, so they can be added block-wise 
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Ay Ap Ap a By By By 
Az) A22 A23 By, Bz. By3 
— {Aut Bu Ant Bi Ais + Bis 
Az, + By, Azz + Baz Az3 + Ba3 
¥  24-~ So fo <3 
= | | 
—7 0 | 6 | 9 8 
3 8 | 2 | 7 -l 
Aji “Ay A Bu 
7. The block matrices A = M - 3) and B = By | have conformal partitions if the B; 
Az, Ag Ay By 


index sets, which form the submatrices Aj = 


Al[q;, Bj] of A, are the same as the 6; index sets, 


which form the submatrices Bj, = B[Bj, 7] of B. For instance, the matrix 


1 | 5 | 3 —1 
Ma An Ai Ais} _ | | 

Az, A222 A23 —3 0 | 1 | 6 1 

2 Me Ny as US BF 
4 -—l1 
3 9 
Bu aS SS 

and the matrix B = | By,| = 5 2 | have conformal partitions, so A and B can be 

Bs) SS 
2 —8 
7 -—l 


multiplied block-wise to form the 3 x 2 matrix 


AB = Ay By + Ai2 Bz) + Ai3 Bs; 
Az By + Az2 Bai + A23 B31 
4 -l 2 - 
[ae 3] : Sree ene aa E zi 
Pl faa) 6: tl ee 
| 2 : 9 = 4 ar 5 Z] ; 7] 
[-2 -—19]+ [25 10}+[-1 —23] D9 9 1239. 
=]|/-12 3 af 2 1 —4 = {12 —44 
n allele 9 
29 +61 20 8 —39 —33 10 36 
Zz «| 
i 2. 3 9 | ) 
8. Let A= 45 | 6 andB= | ___|.- Then A and B have conformal partitions. BA 


| 2 


is defined, but B and A do not have conformal partitions. 
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10.2 Block Diagonal and Block Triangular Matrices 


Definitions: 
A matrix A = [aj] € F"*" is diagonal if aj = 0, for alli A j, 1 <i,j <n. 
A diagonal matrix A = [aj] € F"*" is said to be scalar if aj; = a, for alli, 1 < i < n, and some scalar 
aéF,ie, A=al,. 
A matrix A € F"*" is block diagonal if A as a block matrix is partitioned into submatrices Aj € 
F"*"), so that A = [Aj], ee nj; = n, and Aj = 0, for alli # j,1 < i,j < k. Thus, A = 
Ay 0 -. 0 


0 An « O 
. This block diagonal matrix A is denoted A = diag(Aj,,..., Axx), where Aj; is 


0 O «App 
n; X nj, (or sometimes denoted A = Aj; ®--- @ Axe, and called the direct sum of Aj),..., Axx). 

A matrix A = [aj] ¢ F"*" is upper triangular if aj = 0, for alli > j1 <i, j <n. 

An upper triangular matrix A = [aj] € F"*" is strictly upper triangular if aj = 0 for alli > j, 
l<i,j<n. 

A matrix A € F"*" is lower triangular if aj = 0 for alli < j,1 < i,j < n,ie., if A’ is upper 
triangular. 

A matrix A € F"*" is strictly lower triangular if A’ is strictly upper triangular. 

A matrix is triangular it is upper or lower triangular. 

A matrix A € F”"*" is block upper triangular, if A as a block matrix is partitioned into the submatrices 
Ai € F™*"s, so that A = [Aj], ee nj =n, and Aj = 0, for alli > j,1 < i,j < k,ie., considering 


Ay Az -+: Ark 
; ; ; O An +++ Arg 

the Aj blocks as the entries of A, A is upper triangular. Thus, A=} . . . |, where each 
0 0 Arg 

Aj is nj x nj,and ae n; =n. The matrix A is strictly block upper triangular if Aj; = 0, for alli > j, 


1<i,j <k. 
A matrix A € F"*" is block lower triangular if A’ is block upper triangular. 
A matrix A € F"*" is strictly block lower triangular if A’ is strictly block upper triangular. 
A matrix A = [aj] € F"*” is upper Hessenberg if a; = 0, for alli -2 > j, 1 < i,j <n,ie., A has 


411 412 443 on @in-1 Gin 
421 422. 433 pe 42n-1 42n 
O 432 433 ats a3n—1 43n 
the form A = 
0 0 a An—1n-2 An—1n-1 An—-in 
0 0 ee 0 Ann-1 Ann 


A matrix A = [aj] ¢ F"*" is lower Hessenberg if A! is upper Hessenberg. 


Facts: 


The following facts can be found in [HJ85]. This information is also available in many other standard 
references such as [LT85] or [Mey00]. 
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10. 


11. 


12. 
13. 


14. 


. Let D, D’ € F"*" beany diagonal matrices. Then D+ D’ and DD’ are diagonal, and DD’ = D'D. 


If D = diag(d,,...,d,) is nonsingular, then D7! = diag(1/d,,...,1/d,). 

Let D € F"*" bea matrix such that DA = AD for all A € F”"*". Then D isa scalar matrix. 

If A € F"*" isablock diagonal matrix, so that A = diag(Aj),..., Axx), then tr(A) = aa tr(Aj;), 
det(A) = Tk_, det(Aj;), rank(A) = ss rank(A;;), and 0(A) = 0 (Aj) U--+Uo(Ag). 

Let A € F”*" bea block diagonal matrix, so that A = diag( Aj), Ao. ..., Aye). Then A is nonsingu- 
lar ifand only if Aj; is nonsingular for eachi, 1 <i < k. Moreover, Atl= diag( Aj’, Ay wets Aj’). 
See Chapter 4.3 for information on diagonalizability of matrices. 

Let A € F"*" bea block diagonal matrix, so that A = diag(Aj),..., Ax). Then A is diagonalizable 
if and only if Aj; is diagonalizable for eachi, 1 <i <k. 

If A,B € F"*" are upper (lower) triangular matrices, then A + B and AB are upper (lower) 
triangular. If the upper (lower) triangular matrix A = [aj] is nonsingular, then A7! is upper 
(lower) triangular with diagonal entries 1/ay,...,1/dnn. 


Ay Az +++ Arg 
: O An +++ Arg 

Let A € F"*" be block upper triangular, so that A = . : . |. Then tr(A) = 
0 O «Ag 


yo, tr(Ais), det(A) = TW ,det(Ajj), rank(A) > SO, rank(Ajj), and o(A) = o(An) 
U++-Ua(Age). 

Let A = (Aj) € F”*” bea block triangular matrix (either upper or lower triangular). Then A is 
nonsingular if and only if Aj; is nonsingular for each i, 1 <i < k. Moreover, the n; x nj; diagonal 
block entries of A~! are Aas, for eachi, 1 <i<k. 

Schur’s Triangularization Theorem: Let A € C"*”. Then there is a unitary matrix U € C”*” so that 
U* AU is upper triangular. The diagonal entries of U* AU are the eigenvalues of A. 


Let A € R"*". Then there is an orthogonal matrix V ¢ R"*" so that V’ AV is of upper Hessenberg 


Ay Az +++ Ark 
O Ay +++ Arg ; ; ; 
form} . . . |,Where each A;;, 1 <i < k,is1 x 1or2 x 2. Moreover, when Ajj is 
0 O -: Ag 


1 x 1, the entry of Aj; is an eigenvalue of A, whereas when Aj; is 2 x 2, then Aj; has two eigenvalues 
which are nonreal complex conjugates of each other, and are eigenvalues of A. 

(For more information on unitary triangularization, see Chapter 7.2.) 

Let A = [Aj] € F"*" with |o(A)| = n, where Aj,...,Ax € o(A) are the distinct eigenvalues of 
A. Then there is a nonsingular matrix T € F"*" so that T~! AT = diag(Ai,,..., Axx), where each 
Aj € F"*" is upper triangular with all diagonal entries of Aj; equal to 4;, for 1 <i < k (and 


k 
ja) Ni =n). 


: : A A ‘ 

Let A € F"*" be a block upper triangular matrix, of the form A = ; : i , where Ajj is 
22 

nj X nj,l < i,j < 2, and Pear nj = n. (Note that any block upper triangular matrix can be 


said to have this form.) Let x be an eigenvector of Aj;, with corresponding eigenvalue A, so that 
A\1X = Ax, where x is a (column) vector with n; components. Then the (column) vector with n 


components A is an eigenvector of A with eigenvalue i. Let y be a left eigenvector of Az, with 


corresponding eigenvalue ju, so that yAy2 = yy, where y is a row vector with nz. components. Then 
the (row) vector with n components [0 y] isa left eigenvector of A with eigenvalue ju. 
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Examples: 
13 0 0 0 
2 4 0 0 0 
1. The matrix A= |0 0 5 0 0! isa block diagonal matrix, and the trace, determinant, and 
0 0 0 6 8 
000 7 9 


1 3 


rank can be calculated block-wise, where Aj; = a 


, Ax. = 5, and A33 = 


6 8 
7 | , as tr(A) 


25 = d7}_, tr(Aj;), det(A) = (—2)(5)(—2) = T13_, det(Aj;), and rank(A) = 5 = 7}_, rank(Aj;). 
a bc 
2. Let A= |0 d_ e| e€ F°*3, an upper triangular matrix. If a,d, f are nonzero, then A7! = 
0 0 f 
1 b be-—cd 
a ad adf 
1 e 
0 7 “aE : 
0 0 : 
f 
13 0 0 0 
24 5 0 0 
3. The matrix B= |0 0 O 6. 0} isnot block diagonal. However, B is block upper triangular, 
0 00 0 7 
000 0 8 


5 0 0 


with By = ; ff By = 0, B33 = E i By = RF By = f 1: and By3 = [6 Q]. 


Notice that 4 = rank(B) > yo rank(B;;) =2+0+1=3. 


i=1 


1 2 0 O 
3 4 5 O|. ‘ : 
6 7 8 9/38 not lower triangular, but is lower Hessenberg. 


4. The 4 x 4 matrix 
ki 11 12 #13 


10.3. Schur Complements 


. : g A Ap| . ee : F 
In this subsection, the square matrix A = ee ae is partitioned as a block matrix, where Aj, is 
; 21 22 
nonsingular. 
Definitions: 


The Schur complement of Aj; in A is the matrix A. — An Aj Aj2, sometimes denoted A/ Aj). 


Facts: 
I 0} |Ay Ail |Z —AyAwl | An 0 
1. [Zha99] _ = . 
—A,Ay I] |An Axl |0 I 0 A/Ay 
Au Ay 


2. [Zha99] Let A = 


An Az 
Also, rank(A) = rank(A;,) + rank(A/A),). 


, where Aj, is nonsingular. Then det(A) = det(Aj,)det(A/Aj,). 
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3. [Zha99] Let A = Z = . Then A is nonsingular if and only if both A;; and the Schur 
complement of Aj; in A are nonsingular. 

4. [HJ85] Let A = - ct , where Ajj, Ayo, A/Ai1;, A/Az2, and A are nonsingular. Then 
= (A/ Ax)! —Ay Ap(A/An)7! 

— Ajj An (A/Ag2)! (A/An)" 

5. [Zha99] Let A = ic ne , where Aj), Ay2, A/ Aj;, and A/ Az are nonsingular. An equation re- 
lating the Schur complements of Aj; in Aand Ay) in Ais(A/Ay))~! = Ay] +Aq) Av(A/An)7! Aor 
Ayes 

6. [LT85] Let A = - i , where the k x k matrix Aj, is nonsingular. Then rank(A) = k ifand 


only if A/ Ay, = 0. 


7. [Hay68] Let A = eS ae be Hermitian, where Aj; is nonsingular. Then the inertia of A is 
12 22 
in(A) = in(Aj);) + in(A/Aj)). 
A A i ; : : xe 
8. [Hay68] Let A = he ie. be Hermitian, where Aj; is nonsingular. Then A is positive 
12 22 


(semi)definite if and only if both A); and A/ Aj, are positive (semi)definite. 


Examples: 
Ls 2 33: 
l. LettA=|4 5 6]. Then with A;, = 1, we have 
7 8 10 
1 0 1 0 
4 ; : a =2 ‘ 5 6 4 
_ i — -1 
A Dl gs yoll h ; 4 BE ee 
1 0 
= —3 -6 
. Ee —ll 
bh 5 6 3 -6 
2. LetA=|4 5 6].With A}; = 1,and Ay = 3 fpten A/An = Bo ,A/An = 
7 8 10 


1 
1-—[2 3] ; a 3] = bane 
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1 2 3 
3. LetA= 14 5 6 
7 8 10 


Then, from Fact 5, 


-1 rok 
(A/ Aas) = ( >) =17417[2 3] & | 4 i 


= Ae + Ay! An(A/An) | An Aj. 


10.4 Kronecker Products 


Definitions: 
Let A € F”*" and B € F?*4, Then the Kronecker product (sometimes called the tensor product) of A 


a\,B a\.B aad a\,B 

az,B a7B Peet a7,B 
and B, denoted A @ B, is the mp x nq partitioned matrix A ®@ B = 

41 B 4n2B rg Ann B 


Let A € F”*" and let the jth column of A, namely, A[{1,...,m},{j}] be denoted aj, for 1 < j <n. 
a) 


a2 
The column vector with mn components, denoted vec(A), defined by vec(A) = | . | € F””, is the 


an 
vec-function of A, i.e., vec( A) is formed by stacking the columns of A on top of each other in their natural 
order. 


Facts: 


All of the following facts except those for which a specific reference is given can be found in [LT85]. 


Let Ac F”*" and B € F’*41, Ifa € F, thena(A® B) = (aA) ® B= A® (aB). 

Let A,B € F”™*" andC € F’*1, Then(A+ B)Q@C=A@QC+BOC. 

Let Ac F™*" and B,C € F®“1. Then A@®(B+C)=A@QB+AQC. 

Let Ac F™*", B € F’“4,andC € F’*’. Then A@®(B@C)=(A@®B)@C. 

Let A € F™*" and B € F’*4, Then (A® B)? = A? @ B?. 

[MM64] Let A € C”*" and B € C?*4, Then (A @ B) = AQB. 

[MM64] Let A € C””" and B € C’*4. Then (A @ B)* = A* @ B*. 

Let A € F*", B € F?*91,C € F"*", and D € F1™**. Then (A ® B)(C @ D) = AC @ BD. 
Let Ac F™*" and B € F®“1, Then A® B=(A® Ip)Un ® B)=(Un ® B\(A® I,). 

IfA € F™*"and B € F”"*" are nonsingular, then A@ B isnonsingularand(A@B)~! = A~'@Bo!. 
Let A, Az,:-: , Ay € F™*™, and By, By,--- , Be € F"*". Then (A; @ B,)(A2@ Bz) ++: (Ap@® By) = 
(A, A2--+ Ax) @ (B, Bz +++ Bx). 

. Let Ae F”*" and B € F"™". Then (A @ B)* = Ak @ BK. 


ee 
Be SOOO OY: A SHR Se On 


a 
N 
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13. IfA € F™*" and B € F"*", then there isan mn x mn permutation matrix P sothat P™(A@B)P = 
BOA. 

14. Let A, B € F”*". Then vec(aA + DB) = a vec(A) + b vec(B), for anya,b € F. 

15. If Ae F”*", B € F?*4, and X € F"*?, then vec(AXB) = (B! @ A)vec(X). 

16. IfA € F”™*” and B € F"*", then det(A® B) = (det(A))”(det(B))”, tr(A@ B) = (tr(A))(tr(B)), 
and rank(A ® B) = (rank(A))(rank(B)). 

17. Let A € F”™*™ and B € F"*", with o(A) = {Aj,...,Am} and o(B) = {,...,Un}. Then 
A@Be F™* has eigenvalues {A, f4;|1 < s < m,1 < t < n}. Moreover, if the right eigenvectors 
of A are denoted x;, and the right eigenvectors of B are denoted y;, so that Ax; = A;x; and 
By; = ;yj;, then (A @ B)(x; @ yj) = Aiwj(% ®y;). 

18. Let A € F”*™ and B € F"*", with o(A) = {Aj,...,Am} and o(B) = {,...,U,}. Then 
(I, ® A) + (B @ I) has eigenvalues {A, + w:|1 <s <m,1<t <n}. 

19. Let p(x, y) € F[x, y] so that p(x, y) = = ajjx' y/, where aj¢F,1<i<k,1l<j<k.Let 
Ae F™*™ and B € F"*". Define p(A; B) to be the mn x mn matrix p(A; B) = peer ai( Ai ® 
Bi). Ifo(A) = {A1,...,Am} and o(B) = {1,..., Mn}, then o(p(A; B)) = {p(s ull <s < 
m1l<t <n}. 

20. Let Ay, Ay € F™*™, By, By € F"*". If A; and Ay) are similar, and B, and By, are similar, then 
A, ® B, is similar to A2 ® Bo. 

21. If Ae F™*", B € FP*4, and X € F"*?, then vec(AX) = (Ip @ A)vec(X), vec(XB) = 
(BT @ I,)vec(X), and vec(AX + XB) = [(Ip ® A) + (B? @ I) ]vec(X). 

22. If Ae F™*", B € F?*1,C € F™”4, and X € F"*?, then the equation AX B = C can be written 
in the form (B? @ A)vec(X) = vec(C). 

23. Let Ac F™*", Be F"*",C € F™*" and X € F™*", Then the equation AX + XB = C canbe 
written in the form [(I,, ®@ A) + (B™ @ I,,)}vec(X) = vec(C). 

24. Let Ae C”*” and B € C"*" be Hermitian. Then A ® B is Hermitian. 

25. Let Ae C”*” and B € C"*" be positive definite. Then A @ B is positive definite. 

Examples: 

12 3 -1 -—2 -3 

123 4 5 6 -4 -—-5 -6 

1. Let A= A 4 5 6|.ThnA®B= (jae aes as 

2: 7809 0 00 2 4 6 

0 0 0 8 10. «12 

0 0 0 14 #16 = «18 

2. Let A= ; ye vec(A) = = . 
2 
3. Let Ae F”*” and B € F"*" If A is upper (lower) triangular, then A @ B is block upper (lower) 
triangular. If A is diagonal then A @ B is block diagonal. If both A and B are upper (lower) 
triangular, then A @ B is (upper) triangular. Ifboth A and B are diagonal, then A @ B is diagonal. 
4. Let Ac F”™*" and B € F?*1. If A@ B = 0, then A=O0or B= 0. 
ail, ay), aa Ainly 
ay I, ay I), "be Arnly 2 

5. Let Ae F”™*". Then A@ I, = : . . . e F™*" Let B € FP*?, Then 
AmiIn ant, aoe Amnly 


I, ® B = diag(B, B,...,B) € F"?*"?, and In, @ In = Inn. 
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Matrix functions are used in many areas of linear algebra and arise in numerous applications in science 
and engineering. The most common matrix function is the matrix inverse; it is not treated specifically in 
this chapter, but is covered in Section 1.1 and Section 38.2. This chapter is concerned with general matrix 
functions as well as specific cases such as matrix square roots, trigonometric functions, and the exponential 
and logarithmic functions. 

The specific functions just mentioned can all be defined via power series or as the solution of nonlinear 
systems. For example, cos(A) = I — A?/2!+ A‘/4! — -- -. However, a general theory exists from which a 
number of properties possessed by all matrix functions can be deduced and which suggests computational 
methods. This chapter treats general theory, then specific functions, and finally outlines computational 
methods. 


11.1 General Theory 


Definitions: 


A function of a matrix can be defined in several ways, of which the following three are the most generally 
useful. 


* Jordan canonical form definition. Let A € C"*" have the Jordan canonical form Z~'AZ = J, = 
diag (Ji (41), Ta(A2), 0005 Tp(p)) , where Z is nonsingular, 


Vege : ecm, (11.1) 
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and m, + m2 +++: + mp =n. Then 


f(A) = ZfUJa)Z" = Zdiag( f Ter) Z1, (11.2) 
where 
Fx) f'Ox) pe 
f Uk(Ak)) = fm) -* ; (11.3) 
fx) 
fOr) 
* Polynomial interpolation definition. Denote by 4y,...,A; the distinct eigenvalues of A and let n; be 


the index of A, that is, the order of the largest Jordan block in which 4; appears. Then f(A) := r(A), 
where r is the unique Hermite interpolating polynomial of degree less than }*;_, n; that satisfies 
the interpolation conditions 


rHa)y= fA), f=On;-1, i=lss. (11.4) 


Note that in both these definitions the derivatives in (11.4) must exist in order for f (A) to be defined. 
The function f is said to be defined on the spectrum of A if all the derivatives in (11.4) exist. 


* Cauchy integral definition. 


1 
wi 


f(A): i f(z)(zI — A) dz, (11.5) 


where f is analytic inside a closed contour I that encloses o (A). 


When the function f is multivalued and A has a repeated eigenvalue occurring in more than one Jordan 
block (i.e., A is derogatory), the Jordan canonical form definition has more than one interpretation. Usually, 
for each occurrence of an eigenvalue in different Jordan blocks the same branch is taken for f and its 
derivatives. This gives a primary matrix function. If different branches are taken for the same eigenvalue 
in two different Jordan blocks, then a nonprimary matrix function is obtained. A nonprimary matrix 
function is not expressible as a polynomial in the matrix, and if such a function is obtained from the 
Jordan canonical form definition (11.2) then it depends on the matrix Z. In most applications it is 
primary matrix functions that are of interest. For the rest of this section f(A) is assumed to be a primary 
matrix function, unless otherwise stated. 


Facts: 


Proofs of the facts in this section can be found in one or more of [Hig], [HJ91], or [LT85], unless otherwise 
stated. 


1. The Jordan canonical form and polynomial interpolation definitions are equivalent. Both defini- 
tions are equivalent to the Cauchy integral definition when f is analytic. 

f (A) is a polynomial in A and the coefficients of the polynomial depend on A. 

f(A) commutes with A. 

f(AT) = f(A)? 

For any nonsingular X, f(X AX~!) = Xf(A)X7!. 

If A is diagonalizable, with Z~'AZ = D = diag(d),d),...,d,), then f(A) = Zf(D)Z7 = 
Z diag( f (di), f (d2),-.-, f(dn))Z~. 

7. f (diag( Ar, A2,.--+Am)) = diag( f(A), f(A2),...5 f(Am)). 


OS eS 
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8. Let f and g be functions defined on the spectrum of A. 


(a) Ifh(t) = f(t) + g(t), then h(A) = f(A) + g(A). 
(b) If h(t) = f(t)g(t), then h(A) = f(A)g(A). 


9. Let G(u,,...,u,) be a polynomial in u),...,u,; and let f{,..., f; be functions defined on the 
spectrum of A. If g(A) = G(fi(A),..., f:(A)) takes zero values on the spectrum of A, then 
g(A) = G(fi(A),..., fi(A)) = 0. For example, sin?(A) + cos?(A) = I, (Al/?)? = A, and 
eA = cosA+isin A. 

10. Suppose f has a Taylor series expansion 


22. (k) 
f(z) = >> a(z— a)" (n= i 7 


k=0 


with radius of convergence r. If A € C"*", then f(A) is defined and is given by 


f(A) = D7 a(A— at) 


k=0 
if and only if each of the distinct eigenvalues 4),...,A; of A satisfies one of the conditions: 
(a) |A; -—a] <r. 


(b) |A; —@| =r and the series for f”~!(A), where n; is the index of A;, is convergent at the point 
A= Kio i=lis. 
11. [Dav73], [Des63], [GVL96, Theorem 11.1.3]. Let T € C”*” be upper triangular and suppose that 
f is defined on the spectrum of T. Then F = f(T) is upper triangular with fj; = f(t) and 
hij os eS Bsq.s1 bs1,s2 ++ + Esp_ayse flAso» tee Ase]s 
(Sos-+9Sk) €Sij 
where A; = ;;, Sj; is the set of all strictly increasing sequences of integers that start at i and end at 


jand f[A,,,...,As,] is the kth order divided difference of f atA,,,...,As,- 


Examples: 
1. For A; 4 Aa, 


FAz) = fr) 


(3 sD- [Pe | 


(lo i))=[%o" “fas') 


ForA,; =A,=A, 


2. Compute e4 for the matrix 
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Hence, using the Jordan canonical form definition, we have 


—6+10e -—3+6e -—2+4e 


6 

2 

2, 
6 — 7e 3 — 4e 2 — 3e 
6+ 6e 3+ 3e 2+ 3e 


3. Compute VA for the matrix in Example 2. To obtain the square root, we use the polynomial 
interpolation definition. The eigenvalues of A are 0 and 1, with indices 1 and 2, respectively. The 
unique polynomial r of degree at most 2 satisfying the interpolation conditions r(0) = f(0), 


r(1) = f(Q),r’() = f'(1) is 
r(t) = f(O\(t- 1)? + 4(22—-t)f(1) +t(t- 1) f'(). 


With f(t) = ¢'/?, taking the positive square root, we have r(t) = t(2 — t) + ¢(t — 1)/2 and, 
therefore, 


—6 —3.5 = 
6 3 3 


Al? = A(2I— A)+ A(A—1)/2= | 8 5 3 


4. Consider the m, x my Jordan block J;,(A,) in (11.1). The polynomial satisfying the interpolation 
conditions (11.4) is 


= m-1 
fg) + + —— Fa) 


(m, — 1)! 


(t — Ax)? 
2! 


r(t)= fx) + (t — An) f(g) + 


which, of course, is the first mm, terms of the Taylor series of f about Ax. Hence, from the polynomial 
interpolation definition, 


f De (Ag)) =r Ue (Ak) 
= f(Ag)T + Jean) — Ag) f'n) 4 


(Te(An) — ART)! 
(mp = 1)! 


(e(An) — Akl)? 
2! 


f" (Ax) J... 


ee Coe 


The matrix (Jz (Ax) — AgL)/ is zero except for 1s on the jth superdiagonal. This expression for 
f Ux (Ax)) is, therefore, equal to that in (11.3), confirming the consistency of the first two definitions 
of f(A). 


11.2 Matrix Square Root 


Definitions: 
Let A € C”*". Any X such that X? = A is a square root of A. 
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Facts: 


Proofs of the facts in this section can be found in one or more of [Hig], [HJ91], or [LT85], unless otherwise 
stated. 


1. If Ae C”*" has no eigenvalues on Ro (the closed negative real axis) then there is a unique square 
root X of A each of whose eigenvalues is 0 or lies in the open right half-plane, and it is a primary 
matrix function of A. This is the principal square root of A and is written X = A’/?. If A is real 
then A'/? is real. An integral representation is 


2 2 2 -1 
Al =—A (t*I + A)~ dt. 
Tw 0 

2. A positive (semi)definite matrix A € C”*” has a unique positive (semi)definite square root. (See 
also Section 8.3.) 

3. [CL74] A singular matrix A € C”*” may or may not have a square root. A necessary and sufficient 
condition for A to have a square root is that in the “ascent sequence” of integers d), dz, .. . defined 
by 


d; = dim(ker(A‘)) — dim(ker(A‘~!)), 


no two terms are the same odd integer. 

4. A € R"*" has a real square root if and only if A satisfies the condition in the previous fact and A 
has an even number of Jordan blocks of each size for every negative eigenvalue. 

5. Then x n identity matrix I, has 2” diagonal square roots diag(+1). Only two of these are primary 
matrix functions, namely I and —I. Nondiagonal but symmetric nonprimary square roots of I, 
include any Householder matrix I — 2vv! /(v" v) (v # 0) and the identity matrix with its columns 
in reverse order. Nonsymmetric square roots of I, are easily constructed in the form XDX7, 
where X is nonsingular but nonorthogonal and D = diag(+1) A +I. 


Examples: 


1. The Jordan block E | has no square root. The matrix 
0 1 0 
0 0 O 
0 0 0 
has ascent sequence 2, 1,0,... and so does have a square root — for example, the matrix 


0 0 1 
0 0 0 
0 1 0 


11.3. Matrix Exponential 


Definitions: 
The exponential of A € C”*", written e“ or exp(A), is defined by 
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Facts: 


Proofs of the facts in this section can be found in one or more of [Hig], [HJ91], or [LT85], unless otherwise 
stated. 


1, e(AtB)t — eAteB! holds for all ¢ if and only if AB = BA. 
2. The differential equation in n x n matrices 
dY 
ae AY, Y(0)=C, A,Y ec", 


has solution Y(t) = eC. 
3. The differential equation in n x n matrices 


dY 
Prk AY + YB, Y(0) =C, A,B,Y €C"*", 


has solution Y(t) = e“’Ce®". 

4. A € C’" is unitary if and only if it can be written A = e'%, where H is Hermitian. In this 
representation H can be taken to be Hermitian positive definite. 

5. A € R"™" is orthogonal with det(A) = 1 if and only if A = e° with S € R”™*" skew-symmetric. 


Examples: 


1. Fact 5 is illustrated by the matrix 


for which 


11.4 Matrix Logarithm 


Definitions: 
Let A € C”*". Any X such that e* = A isa logarithm of A. 


Facts: 


Proofs of the facts in this section can be found in one or more of [Hig], [HJ91], or [LT85], unless otherwise 
stated. 


1. If A has no eigenvalues on R-, then there is a unique logarithm X of A all of whose eigenvalues lie 
in the strip {z: —a < Im(z) < z }. Thisis the principal logarithm of A and is written X = log A. 
If A is real, then log A is real. 

2. If p(A) < 1, 


log(I + A A as 

og + A)=A- 42S 4. 

3. A € R"*" has a real logarithm if and only if A is nonsingular and A has an even number of Jordan 
blocks of each size for every negative eigenvalue. 

4. exp(log A) = A holds when log is defined on the spectrum of A € C”*”. But log(exp(A)) = A 
does not generally hold unless the spectrum of A is restricted. 

5. If A € C”*” is nonsingular then det(A) = exp(tr(log A)), where log A is any logarithm of A. 
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Examples: 


For the matrix 
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11141 
Aes 0 1 2 3 
0 0 1 3 
0 0 0 1 
we have 
0 1 0 0 
0 0 2 0 
los) =)9 9 0 3 
0 0 0 0 
11.5 Matrix Sine and Cosine 
Definitions: 
The sine and cosine of A € C”*” are defined by 
A’ (=1) x 
cos(A) =I T + t Qk)! A™ +++, 
A (ae 
A)=A-——+-:- tt 
sin( A) 3 fore Qk + DI + 


Facts: 


Proofs of the facts in this subsection can be found in one or more of [Hig], [HJ91], or [LT85], unless 


otherwise stated. 


cos(2A) = 2cos?(A) — I. 
sin(2A) = 2 sin(A) cos(A). 
cos?(A) + sin?(A) = I. 
The differential equation 


eit: ie 


ey ae, 
dt? ier 


has solution 


y(t) = cos(W At) yo + (VA) ~ sin(V At) yi, 


where JA denotes any square root of A. 
Examples: 


1. For 


we have 


ef = 


cos a 
i sina 


y(0) = Yoo 


i sina 
cos a 


y'(0) = % 


ip 
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2. For 
1 1 1 1 
0 -1 —2 -3 
si 0 Le 32) 
0 0 0 -l 
we have 
sin(1) — sin(1) sin(1) sin(1) 
F 0 —sin(1) —2sin(1) —3sin(1) 
A)= 1)J, A)= : : 

COCA) cok!) a) 0 0 sin(1) 3 sin(1) 
0 0 0 — sin(1) 


and sin?(A) = sin(1)?I, so cos(A)? + sin(A)* = I. 


11.6 Matrix Sign Function 


Definitions: 


If A = ZJ4Z~' € C"*" isa Jordan canonical form arranged so that 


yp 0 
i=| ; 72> 


where the eigenvalues of J wv € C?*? lie in the open left half-plane and those of J . € C1*4 lie in the 
open right half-plane, with p + q = n, then 
sign(A) = Z|? 9 | 271 
Bee Ws, Sipiio= 2 
Alternative formulas are 
sign(A) = A(A?)-?, (11.6) 


2 CO 
jens “af (21+ a2)" dt. 
0 


If A has any pure imaginary eigenvalues, then sign(A) is not defined. 


Facts: 


Proofs of the facts in this section can be found in [Hig]. 
Let S = sign(A) be defined. Then 


S? = I (S is involutory). 

S is diagonalizable with eigenvalues +1. 

SA= AS. 

If A is real, then S is real. 

If A is symmetric positive definite, then sign(A) = I. 


ty ee on 


Examples: 


1. For the matrix A in Example 2 of the previous subsection we have sign(A) = A, which follows 
from (11.6) and the fact that A is involutory. 
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11.7 Computational Methods for General Functions 


Many methods have been proposed for evaluating matrix functions. Three general approaches of 
wide applicability are outlined here. They have in common that they do not require knowledge of 
Jordan structure and are suitable for computer implementation. References for this subsection are 
[GVL96], [Hig]. 


1. Polynomial and Rational Approximations: 


Polynomial approximations 
Pm(X) = 5" OX*, =e EC, XEC™, 
k=0 


to matrix functions can be obtained by truncating or economizing a power series representation, or by 
constructing a best approximation (in some norm) of a given degree. How to most efficiently evaluate a 
polynomial at a matrix argument is a nontrivial question. Possibilities include Horner’s method, explicit 
computation of the powers of the matrix, and a method of Paterson and Stockmeyer [GVL96, Sec. 11.2.4], 
[PS73], which is a combination of these two methods that requires fewer matrix multiplications. 

Rational approximations r4(X) = Pm(X)qx(X)~! are also widely used, particularly those arising 
from Padé approximation, which produces rationals matching as many terms of the Taylor series of 
the function at the origin as possible. The evaluation of rationals at matrix arguments needs careful 
consideration in order to find the best compromise between speed and accuracy. The main possibilities 
are 


* Evaluating the numerator and denominator polynomials and then solving a multiple right-hand 
side linear system. 

¢ Evaluating a continued fraction representation (in either top-down or bottom-up order). 

¢ Evaluating a partial fraction representation. 


Since polynomials and rationals are typically accurate over a limited range of matrices, practical methods 
involve a reduction stage prior to evaluating the polynomial or rational. 


2. Factorization Methods: 


Many methods are based on the property f(XAX~') = Xf(A)X7!. If X can be found such that 
B = XAX™~ has the property that f(B) is easily evaluated, then an obvious method results. When A 
is diagonalizable, B can be taken to be diagonal, and evaluation of f(B) is trivial. In finite precision 
arithmetic, though, this approach is reliable only if X is well conditioned, that is, if the condition number 
K(X) = ||X|||| X71]| is not too large. Ideally, X will be unitary, so that in the 2-norm «)(X) = 1. For 
Hermitian A, or more generally normal A, the spectral decomposition A = QDQ* with Q unitary and 
D diagonal is always possible, and if this decomposition can be computed then the formula f(A) = 
Qf (D) Q* provides an excellent way of computing f(A). 

For general A, if X is restricted to be unitary, then the furthest that A can be reduced is to Schur form: 
A = QTQ*, where Q is unitary and T upper triangular. This decomposition is computed by the QR 
algorithm. Computing a function of a triangular matrix is an interesting problem. While Fact 11 of section 
11.1 gives an explicit formula for F = f(T), the formula is not practically viable due to its exponential cost 
inn. Much more efficient is a recurrence of Parlett [Par76]. This is derived by starting with the observation 
that since F is representable as a polynomial in T, F is upper triangular, with diagonal elements f(t; ). 
The elements in the strict upper triangle are determined by solving the equation FT = TF. Parlett’s 
recurrence is: 
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Algorithm 1. Parlett’s recurrence. 


fii = f(ti),t = lin 
for j = 2:n 
fori = j —1:-1:1 


be Dae PR en MeN is 
fig = hj poe fale oe fixtkj tik fj (Gi tjj) 


di k=i+1 
end 
end 
This recurrence can be evaluated in 2n*/3 operations. The recurrence breaks down when tj, = tj 


for some i # j. In this case, T can be regarded as a block matrix T = (Tj), with square diagonal 
blocks, possibly of different sizes. T can be reordered so that no two diagonal blocks have an eigenvalue in 
common; reordering means applying a unitary similarity transformation to permute the diagonal elements 
whilst preserving triangularity. Then a block form of the recurrence can be employed. This requires the 
evaluation of the diagonal blocks F;; = f(Tj;), where T;; will typically be of small dimension. A general 
way to obtain Fj; is via a Taylor series. The use of the block Parlett recurrence in combination with a Schur 
decomposition represents the state of the art in evaluation of f(A) for general functions [DH03]. 


3. Iteration Methods: 


Several matrix functions f can be computed by iteration: 
Xk = g(Xx), Xo = A, (11.7) 


where, for reasons of computational cost, g is usually a polynomial or a rational function. Such an iteration 
might converge for all A for which f is defined, or just for a subset of such A. A standard means of deriving 
matrix iterations is to apply Newton’s method to an algebraic equation satisfied by f(A). The iterations 
most used in practice are quadratically convergent, but iterations with higher orders of convergence are 
known. 

4. Contour Integration: 

The Cauchy integral definition (11.5) provides a way to compute or approximate f(A) via contour 
integration. While not suitable as a practical method for all functions or all matrices, this approach can 
be effective when numerical integration is done over a suitable contour using the repeated trapezium 


rule, whose high accuracy properties for periodic functions integrated over a whole period are beneficial 
[DH05], [TW05]. 


11.8 Computational Methods for Specific Functions 


Some methods specialized to particular functions are now outlined. References for this section are 
[GVL96], [Hig]. 


1. Matrix Exponential: 


A large number of methods have been proposed for the matrix exponential, many of them of pedagogic 
interest only or of dubious numerical stability. Some of the more computationally useful methods are 
surveyed in [MVL03]. Probably the best general-purpose method is the scaling and squaring method. In 
this method an integral power of 2, 0 = 2° say, is chosen so that A/o has norm not too far from 1. The 
exponential of the scaled matrix is approximated by an [m/m] Padé approximant, e“/ 2 tmm(A/25), 
and then s repeated squarings recover an approximation to e4:e4 © rmm_(A/2°)* . Symmetries in the Padé 
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approximant permit an efficient evaluation of r,,(A). The scaling and squaring method was originally 
developed in [MVL78] and [War77], and it is the method employed by MATLAB’s expm function. How 
best to choose o and m is described in [Hig05]. 


2. Matrix Logarithm: 


The (principal) matrix logarithm can be computed using an inverse scaling and squaring method based 
on the identity log A = 2* log Al!’ 2 where A is assumed to have no eigenvalues on R~. Square roots 
are taken to make || A!/2 — || small enough that an [m/m] Padé approximant approximates log Al/ a 
sufficiently accurately, for some suitable m. Then log A is recovered by multiplying by 2*. To reduce the 
cost of computing the square roots and evaluating the Padé approximant, a Schur decomposition can be 
computed initially so that the method works with a triangular matrix. For details, see [CHKLO1], [Hig01], 
or [KL89, App. A]. 


3. Matrix Cosine and Sine: 


A method analogous to the scaling and squaring method for the exponential is the standard method 
for computing the matrix cosine. The idea is again to scale A to have norm not too far from 1 and then 
compute a Padé approximant. The difference is that the scaling is undone by repeated use of the double- 
angle formula cos(2A) = 2 cos? A—I, rather than by repeated squaring. The sine function can be obtained 
as sin(A) = cos(A — 3-1). (See [SB80], [HS03], [HH05].) 

4. Matrix Square Root: 


The most numerically reliable way to compute matrix square roots is via the Schur decomposition, 
A = QT Q [BH83]. Rather than use the Parlett recurrence, a square root U of the upper triangular factor 
T can be computed by directly solving the equation U? = T. The choices of sign in the diagonal of U, 
uj; = s/t, determine which square root is obtained. When A is real, the real Schur decomposition can 
be used to compute real square roots entirely in real arithmetic [Hig87]. 

Various iterations exist for computing the principal square root when A has no eigenvalues on R~. The 
basic Newton iteration, 


1 -1 
Xi = 5 (Xk +X," A), Xo = A, (11.8) 


is quadratically convergent, but is numerically unstable unless A is extremely well conditioned and its use 
is not recommended [Hig86]. Stable alternatives include the Denman—Beavers iteration [DB76] 
Xk = (X. +Y,"); Xo = A, 


Yea = (Y%. + X;"), Y= ZI, 


Nl NIe 


for which limg_,o. Xp = A!/? and limy_, 5 Y; = A~!/, and the Meini iteration [Mei04] 


Yer = —YeZe Yes Y=I-A, 
Zea = Ze+ 2Ve415 Zo = 21 + A), 


for which ¥, > 0 and Z;, — 4A!/?. Both of these iterations are mathematically equivalent to (11.8) and, 
hence, are quadratically convergent. 
An iteration not involving matrix inverses is the Schulz iteration 


Yeu = 5¥e(31 — Zp¥;), Yo= A, 
Zev = 5031 — ZeVu)Ze, Zo =I, 


for which Y, > A!/? and Z, > A7~'/? quadratically provided that || diag(A — I, A—I)|| < 1, where the 
norm is any consistent matrix norm [Hig97]. 
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5. Matrix Sign Function: 


The standard method for computing the matrix sign function is the Newton iteration 
l -1 
Xi = 5 (Xk +X"), Xo= A, 


which converges quadratically to sign(A), provided A has no pure imaginary eigenvalues. In practice, a 
scaled iteration 


1 eee 
Xt = 7 (oR Xk +u;,'X;') Xo=A 


is used, where the scale parameters jz; are chosen to reduce the number of iterations needed to enter the 
regime where asymptotic quadratic convergence sets in. (See [Bye87], [KL92].) 
The Newton-Schulz iteration 


1 
Xia = 5 Xe(31 — Xi), Xo=A, 


involves no matrix inverses, but convergence is guaranteed only for || — A?|| < 1. 
A Padé family of iterations 


Xkt1 = Xk Pom (1 — X7) em (1— x2)", Xyo=A 


is obtained in [KL91], where pem(€)/qem(é) is the [€/m] Padé approximant to (1 — €)~!/?. The iteration 
is globally convergent to sign(A) for € = m— 1 and ¢ = m, and for £ > m — 1 is convergent when 
|| — A?|| < 1, with order of convergence £ + m+ 1 inall cases. 
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Bilinear forms are maps defined on V x V, where V is a vector space, and are linear with respect to each of 
their variables. There are some similarities between bilinear forms and inner products that are discussed 
in Chapter 5. Basic properties of bilinear forms, symmetric bilinear forms, and alternating bilinear forms 
are discussed. The latter two types of forms satisfy additional symmetry conditions. 

Quadratic forms are obtained from symmetric bilinear forms by equating the two variables. They are 
widely used in many areas. A canonical representation of a quadratic form is given when the underlying 
field is R or C. 

When the field is the complex numbers, it is standard to expect the form to be conjugate linear rather 
than linear in the second variable; such a form is called sesquilinear. The role of a symmetric bilinear 
form is played by a Hermitian sesquilinear form. The idea of a sesquilinear form can be generalized to 
an arbitrary automorphism, encompassing both bilinear and sesquilinear forms as y-sesquilinear forms, 
where g is an automorphism of the field. 

Quadratic, bilinear, and g-sesquilinear forms have applications to classical matrix groups. (See Chapter 
67 for more information.) 


12.1 Bilinear Forms 


It is assumed throughout this section that V is a finite dimensional vector space over a field F. 
Definitions: 
A bilinear form on V is a map f from V x V into F which satisfies 
f (au; + bu,v) =af(w,v)+b0f(w,v), u,w,vEe V, abe F, 
and 


f(u,av, + bv2) =af(uvi)+bf(uv2), wv,wEV, abe F. 
12-1 


12-2 Handbook of Linear Algebra 


The space of all bilinear forms on V is denoted B(V, V, F). 

Let B = (wi, W2,... Wn) be an ordered basis of V and let f € B(V, V, F). The matrix representing 
f relative to B is the matrix A = [ajj] € F"*" such that aj; = f (wi, Wj). 

The rank of f € B(V,V, F), rank( f), is rank(A), where A is a matrix representing f relative to an 
arbitrary ordered basis of V. 

f € B(V, V, F) isnondegenerate ifits rank is equal to dim V, and degenerate if it is not nondegenerate. 

Let A, B € F"*". B is congruent to A if there exists an invertible P € F"*” such that B = PTAP. 

Let f,g € B(V, V, F). g is equivalent to f if there exists an ordered basis B of V such that the matrix 
of g relative to B is congruent to the matrix of f relative to B. 

Let T bea linear operator on V and let f € B(V,V,F). T preserves f if f(Tu, Tv) = f(u,v) for all 
uve V. 


Facts: 
Let f € B(V, V, F). The following facts can be found in [HK71, Chap. 10]. 


1. f isa linear functional in each of its variables when the other variable is held fixed. 
2. Let B = (wy, W2,... ; Wn) be an ordered basis of V and let 


n n 
u= > aj Wi» v= > bw. 
i=l i=l 
Then, 


f(uv) = 5° S° ajb; f(wisw)). 
i=1 j=l 
3. Let A denote the matrix representing f relative to B, and let [u] and [v] be the vectors in F” that 
are the coordinate vectors of u and v, respectively, with respect to B. Then f(u,v) = [u] 4 A[v] B: 
4. Let B and B’ be ordered bases of V, and P be the matrix whose columns are the B-coordinates of 
vectors in 8B’. Let f € B(V,V,F). Let A and B denote the matrices representing f relative to B 
and B’. Then 


B= PTAP. 


5. The concept of rank of f, as given, is well defined. 

6. ThesetL = {ve V: f(u,v) = 0 for all u € V} isa subspace of V and rank( f) = dim V—dim L. 
In particular, f is nondegenerate if and only if L = {0}. 

7. Suppose that dim V = n. The space B(V, V, F) isa vector space over F under the obvious addition 
of two bilinear forms and multiplication of a bilinear form by a scalar. Moreover, B(V, V, F) is 
isomorphic to F”*". 

8. Congruence is an equivalence relation on F"*”. 

9. Let f € B(V, V, F) be nondegenerate. Then the set of all linear operators on V, which preserve 
f, is a group under the operation of composition. 


Examples: 
1. Let Ac F"*". The map f : F” x F" — F defined by 


n 


n 
f(u,v) = uw’ Av = Soo aijuiv;, u,veé F”, 


i=1 j=l 


is a bilinear form. Since f(e,ej) = ajj, 1,7 = 1,2,...,n, f is represented in the standard basis 
of F" by A. It follows that rank( f) = rank(A), and f is nondegenerate if and only if A is invertible. 
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2. Let C € F”*” and rank(C) = k. The map f : F™*" x F”*" —> F defined by f(A, B) = 
tr(A’CB) is a bilinear form. This follows immediately from the basic properties of the trace 
function. To compute rank( f), let L be defined as in Fact 6, that is, L = {B ¢ F™*": tr(ATCB) = 
0 for all A € F”*"}. It follows that L = {B € F™*" : CB = 0}, which implies that dim L = 
n(m — k). Hence, rank( f) = mn — n(m — k) = kn. In particular, f is nondegenerate if and only 
if C is invertible. 

3. Let R[x;] denote the space of all real polynomials of the form )7j_y a;x'. Then f (p(x), q(x)) = 
p(0)q(0) + p(1)q(1) + p(2)q(2) isa bilinear form on R[x; ]. It is nondegenerate if n = 2 and 
degenerate if n > 3. 


12.2 Symmetric Bilinear Forms 


It is assumed throughout this section that V is a finite dimensional vector space over a field F. 


Definitions: 
Let f € B(V,V, F). Then f is symmetric if f(u,v) = f(v,u) forall u,v € V. 

Let f be a symmetric bilinear form on V, and let u, v € V; uand v are orthogonal with respect to f if 
f (u,v) = 0. 

Let f beasymmetric bilinear form on V. The quadratic form corresponding to f isthe map g : V > F 
defined by g(v) = f(v,v), ve V. 

A symmetric bilinear form f on a real vector space V is positive semidefinite (positive definite) if 
f(v,v) > 0 forallv € V (f(v,v) > 0 foralO Ave V). 

f is negative semidefinite (negative definite) if — f is positive semidefinite (positive definite). 

The signature of a real symmetric matrix A is the integer 2 — v, where (z, v, 5) is the inertia of A. (See 
Section 8.3.) 

The signature of a real symmetric bilinear form is the signature of a matrix representing the form 
relative to some basis. 


Facts: 


Additional facts about real symmetric matrices can be found in Chapter 8. Except where another reference 
is provided, the following facts can be found in [Coh89, Chap. 8], [HJ85, Chap. 4], or [HK71, Chap. 10]. 


1. A positive definite bilinear form is nondegenerate. 

2. An inner product on a real vector space is a positive definite symmetric bilinear form. Conversely, 
a positive definite symmetric bilinear form on a real vector space is an inner product. 

3. Let B be an ordered basis of V and let f € B(V, V, F). Let A be the matrix representing f relative 
to B. Then f is symmetric if and only if A is a symmetric matrix, that is, A= A’. 

4. Let f be a symmetric bilinear form on V and let g be the quadratic form corresponding to f. 
Suppose that the characteristic of F is not 2. Then f can be recovered from g: 


f(uyv) = +[g(ut v) —g(u)—g(v)] foralluve V. 


5. Let f be a symmetric bilinear form on V and suppose that the characteristic of F is not 2. Then 
there exists an ordered basis B of V such that the matrix representing f relative to it is diagonal; 
ie. if A € F”*” is a symmetric matrix, then A is congruent to a diagonal matrix. 

6. Suppose that V isa complex vector space and f isa symmetric bilinear form on V. Letr = rank(f). 
Then there is an ordered basis B of V such that the matrix representing f relative to B is I, © 0. 
In matrix language, this fact states that if A € C”*" is symmetric with rank(A) = 1, then it is 
congruent to I, © 0. 

7. The only invariant of n x n complex symmetric matrices under congruence is the rank. 

8. Two complex n x n symmetric matrices are congruent if and only if they have the same rank. 
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(Sylvester’s law of inertia for symmetric bilinear forms) Suppose that V is a real vector space and 
f is a symmetric bilinear form on V. Then there is an ordered basis B of V such that the matrix 
representing f relative to it has the form I, © —I, © 05. Moreover, z, v, and 5 do not depend on 
the choice of B, but only on f. 


10. (Sylvester’s law of inertia for matrices) If A € R"*" is symmetric, then A is congruent to the diagonal 
matrix D = I, ® —I, @ 05, where (z, v,5) = in(A). 

11. There are exactly two invariants of n x n real symmetric matrices under congruence, namely the 
rank and the signature. 

12. Two real n x n symmetric matrices are congruent if and only if they have the same rank and the 
same signature. 

13. The signature of a real symmetric bilinear form is well defined. 

14. Two real symmetric bilinear forms are equivalent if and only if they have the same rank and the 
same signature. 

15. [Hes68] Let n > 3 and let A, B € R”*” be symmetric. Suppose that x € R”, x! Ax = x! Bx = 
0 => x=0.Thenda,b € Rsuchthat aA +bB is positive definite. 

16. The group of linear operators preserving the form f(u,v) = 5 >/_, u;v; on R" is the real 
n-dimensional orthogonal group, while the group preserving the same form on C” is the com- 
plex n-dimensional orthogonal group. 

Examples: 

1. Consider Example 1 in section 12.1. The map f is a symmetric bilinear form if and only if A = A’. 

The quadratic form g corresponding to f is given by 

n n 
g(a) =S°S- ajjuju;, uc FF”, 
i=1 j=l 
2. Consider Example 2 in section 12.1. The map f isa symmetric bilinear form ifand only ifC = C’. 
3. The symmetric bilinear form f, on R? given by 
fa(u, Vv) =u) V) — 2u;v2 — 2u.v; + au2v2, UWVE R*, acéRisa parameter, 

is an inner product on R? if and only ifa > 4. 

4, Since we consider in this article only finite dimensional vector spaces, let V be any finite dimensional 
subspace of C[0,1], the space of all real valued, continuous functions on [0,1]. Then the map 
f:Vx V => R defined by 

1 
f (u,v) =a Pu(t)v(t)dt, wveV, 
0 
is a symmetric bilinear form on V. 
Applications: 
1. Conic sections: Consider the set of points (x1, x2) in R?, which satisfy the equation 


ane + bx1x2 + ex; + dx, +exm+ f =0, 


where a, b,c,d,e, f € R. The solution set is a conic section, namely an ellipse, hyperbola, parabola, 

or a degenerate form of those. The analysis of this equation depends heavily on the quadratic form 
a b/2 

b/2 ¢ | . If the 

solution of the quadratic equation above represents a nondegenerate conic section, then its type is 

determined by the sign of 4ac — b*. More precisely, the conic is an ellipse, hyperbola, or parabola 

if 4ac — b? is positive, negative, or zero, respectively. 


ax? + bxx2 + cx}, which is represented in the standard basis of R? by A = 
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2. Theory of small oscillations: Suppose a mechanical system undergoes small oscillations about 


an equilibrium position. Let x),x2,...,X, denote the coordinates of the system, and let x = 
(x1, X25... Xn). Then the kinetic energy of the system is given by a quadratic form (in the velocities 
1 X2,.06 Xp) 5x7 Ax, where A is a positive definite matrix. If x = 0 is the equilibrium position, 


then the potential energy of the system is given by another quadratic form +x’ Bx, where B = BT’. 
The equations of motion are Ax + Bx = 0. It is known that A and B can be simultaneously 
diagonalized, that is, there exists an invertible P € R”*" such that P TAP and P! BP are diagonal 
matrices. This can be used to obtain the solution of the system. 


12.3. Alternating Bilinear Forms 


It is assumed throughout this section that V is a finite dimensional vector space over a field F. 


Definitions: 
Let f € B(V,V, F). Then f is alternating if f(v,v) = 0 for allv € V. f is antisymmetric if f(u,v) = 
— f(v,u) forallu,v e V. 

Let A € F"*", Then A is alternating if aj; = 0, i = 1,2,... ,nandajj = —ajj, 1<i<j<n. 


Facts: 
The following facts can be found in [Coh89, Chap. 8], [HK71, Chap. 10], or [Lan99, Chap. 15]. 


1. Let f € B(V, V, F) be alternating. Then f is antisymmetric because for all u,v € V, 
f(u,v) + f(v,u) = f(ut+v,u+v) — f(u,u) — f(v,v) = 0. 


The converse is true if the characteristic of F is not 2. 

2. Let A € F”*" be an alternating matrix. Then A? = —A. The converse is true if the characteristic 
of F is not 2. 

3. Let B be an ordered basis of V and let f € B(V, V, F). Let A be the matrix representing f relative 
to B. Then f is alternating if and only if A is an alternating matrix. 

4. Let f be an alternating bilinear form on V and let r = rank( f). Then r is even and there exists an 
ordered basis B of V such that the matrix representing f relative to it has the form 


0 1 0 1 0 1 
® @:--® @ 0. 
—-1 0 —-1 0 —1 0 
r/2 — times 
There is an ordered basis 6; where f is represented by the matrix - ® 0. 
~ 47/2 


5. Let f € B(V, V, F) and suppose that the characteristic of F is not 2. Define: 
fi: VxV—-F by fi(uv) = 5 [f(uv) + f(v,u)], uve V, 
fh: VxV—->F by fo(uyv) = 5 [f(uv) — f(v,u)], wve V. 
Then f; (f2) is a symmetric (alternating) bilinear form on V, and f = fi + f2. Moreover, this 
representation of f as a sum of a symmetric and an alternating bilinear form is unique. 
6. Let A € F”*" be an alternating matrix and suppose that A is invertible. Then n is even and A 
0 ny 
—Inj 9 
in n(n — 1)/2 variables, called the Pfaffian, such that det(A) = a”, where a € F is obtained by 
substituting into the Pfaffian the entries of A above the main diagonal for the indeterminates. 


is congruent to the matrix , so det(A) is a square in F. There exists a polynomial 
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7. Let f be an alternating nondegenerate bilinear form on V. Then dim V = 2m for some positive 
integer m. The group of all linear operators on V that preserve f is the symplectic group. 


Examples: 
1. Consider Example 1 in section 12.1. The map f is alternating if and only if the matrix A is an 
alternating matrix. 
2. Consider Example 2 in section 12.1. The map f is alternating if and only if C is an alternating 
matrix. 
3. LetC € F"*", Define f : F"*" > F"*" by f(A, B) = tr(ACB — BCA). Then f is alternating. 


12.4 w-Sesquilinear Forms 


This section generalizes Section 12.1, and is consequently very similar. This generalization is required by 
applications to matrix groups (see Chapter 67), but for most purposes such generality is not required, and 
the simpler discussion of bilinear forms in Section 12.1 is preferred. It is assumed throughout this section 
that V is a finite dimensional vector space over a field F and ¢ is an automorphism of F. 


Definitions: 


A g-sesquilinear form on V is a map f: V x V > F, which is linear as a function in the first variable 
and g-semilinear in the second, i.e., 


f (au; + bu,v) =af(uy,v)+bf(w,v), u,w,vEe V, abe F, 
and 
f(u,av, + bv2) = g(a) flu) + ¢(b) f(u,v2), Wwvi,v€ V;, abe F. 


In the case F = C and ¢ is complex conjugation, a g-sesquilinear form is called a sesquilinear form. 
The space of all g-sesquilinear forms on V is denoted B(V, V, F,¢). 

Let B = (wy, W2,... Wn) be an ordered basis of V and let f € B(V, V, F, y). The matrix representing 
f relative to B is the matrix A = [a;;] € F"*" such that aj; = f (wi, Wj). 

The rank of f € B(V, V, F,@), rank( f), is rank(A), where A is a matrix representing f relative to an 
arbitrary ordered basis of V. 

f € B(V,V,F,q@) is nondegenerate if its rank is equal to dim V, and degenerate if it is not 
nondegenerate. 

Let A = [ajj] € F"*". p(A) is the n x n matrix whose i, j-entry is g(a;;). 

Let A, B € F"*". B isy-congruent to A if there exists an invertible P € F"*" such that B = P? Ag(P). 

Let f.g € B(V, V, F,@). g is p-equivalent to f if there exists an ordered basis 6 of V such that the 
matrix of g relative to B is p-congruent to the matrix of f relative to B. 

Let T be a linear operator on V and let f € B(V, V, F,@). T preserves f if f(Tu, Tv) = f(u,v) for 
allu,ve V. 


Facts: 


Let f € B(V, V, F, ¢). The following facts can be obtained by obvious generalizations of the proofs of the 
corresponding facts in section 12.1; see that section for references. 


1. A bilinear form is a g-sesquilinear form with the automorphism being the identity map. 
2. Let B = (wy, W2,... ;Wn) be an ordered basis of V and let 


n n 
u= > aiwig V= > b;Wie 
i=l i=l 
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Then, 


f(y) = Sy awbiF f (wi, Wj). 


i=l j=l 


3. Let A denote the matrix representing the g-sesquilinear f relative to B, and let [u], and [v]g be 
the vectors in F”, which are the coordinate vectors of u and v, respectively, with respect to B. Then 
f (u,v) = [u]ZAg([v]z). 

4. Let B and B’ be ordered bases of V, and P be the matrix whose columns are the B-coordinates of 
vectors in 6’. Let f € B(V, V, F,g). Let A and B denote the matrices representing f relative to B 
and 6’. Then 


B= P'Ag(P). 


5. The concept of rank of f, as given, is well defined. 

6. Theset L = {ve V: f(u,v) = 0 for all u € V} isa subspace of V and rank( f) = dim V—dim L. 
In particular, f is nondegenerate if and only if L = {0}. 

7. Suppose that dim V = n. The space B(V, V, F,@) is a vector space over F under the obvious 
addition of two g-sesquilinear forms and multiplication of a g-sesquilinear form by a scalar. 
Moreover, B(V, V, F,@) is isomorphic to F"*". 

8. g-Congruence is an equivalence relation on F”*”. 

9. Let f € B(V, V, F,@) be nondegenerate. Then the set of all linear operators on V which preserve 
f isa group under the operation of composition. 


Examples: 


1. Let F = Q(/5) = {a + bo/5:.a,b € Q} and gla + b./5) = a — b4/5. Define the y-sesquilinear 
form f on F? by f(u,v) = ug(v). f([1 + 75,3)", [-2V5, -1 + V5]7) = (1 + V5) (2/5) + 
-1-J/5)=7-¥V5. 
The matrix of f with respect to the standard basis is the identity matrix, rank f = 2, and f is 
nondegenerate. 
2. Let Ae F"*". The map f:F"” x F" —> F defined by 


f(uv) =u! Ag(v) = Sn eioes ) uve F", 


i=l j=l 


is a y-sesquilinear form. Since f(e,ej) = ajj, i,j = 1,2,...,n, f isrepresented in the standard 
basis of F” by A. It follows that rank( f) = rank(A), and f is nondegenerate if and only if A is 
invertible. 


12.5 Hermitian Forms 


This section closely resembles the results related to symmetric bilinear forms on real vector spaces. We 
assume here that V is a finite dimensional complex vector space. 


Definitions: 


A Hermitian form on V isa map f :V x V — C, which satisfies 
f(au; + bw,v) =af(w,v)+bf(w,v), wveV, abec, 


and 


f(v,u) = f (u,v), uve V. 
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A Hermitian form f on V is positive semidefinite (positive definite) if f(v,v) > 0 for allv € V 
(f(v,v) > 0 forall0 Ave V). 

f is negative semidefinite (negative definite) if — f is positive semidefinite (positive definite). 

The signature of a Hermitian matrix A is the integer 7 — v, where (z, v,6) is the inertia of A. (See 
Section 8.3.) 

The signature of a Hermitian form is the signature of a matrix representing the form. 

Let A, B € C"*". B is *congruent to A if there exists an invertible S € C”*” such that B = S*AS 
(where S* denotes the Hermitian adjoint of S). 

Let f,g be Hermitian forms on a finite dimensional complex vector space V. g is *equivalent to f if 
there exists an ordered basis B of V such that the matrix of g relative to B is *congruent to the matrix of 
f relative to B. 


Facts: 


Except where another reference is provided, the following facts can be found in [Coh89, Chap. 8], [HJ85, 
Chap. 4], or [Lan99, Chap. 15]. Let f be a Hermitian form on V. 


1. A Hermitian form is sesquilinear. 
2. A positive definite Hermitian form is nondegenerate. 
3. f isa linear functional in the first variable and conjugate linear in the second variable, that is, 


f(u, av, + bvz) = a4 f(uyvy) + b f (u, va). 


4. f(v,v) € Rforallve V. 

5. An inner product on a complex vector space is a positive definite Hermitian form. Conversely, a 
positive definite Hermitian form on a complex vector space is an inner product. 

6. (Polarization formula) 


4f(uv) = f(ut+v,ut+v)— f(u—v,u—v)+ 
+if(u+iv,u-+ iv) —if(u—iv,u— iv). 


7. Let B = (wy, W2,... ;Wn) be an ordered basis of V and let 


u= > aiWin V= 3 biwi- 
i=l i=l 
Then 
f(u,v) = ss Scr 
i=1 j=l 
8. Let A denote the matrix representing f relative to the basis B. Then 
f(uy) = [ug Alves. 


9. The matrix representing a Hermitian form f relative to any basis of V is a Hermitian matrix. 
10. Let A, B be matrices that represent f relative to bases 6 and B’ of V, respectively. Then B is 
*congruent to A. 
11. (Sylvester’s law of inertia for Hermitian forms, cf. 12.2) There exists an ordered basis B of V such 
that the matrix representing f relative to it has the form 


Ix ® —I, ®@ 05. 


Moreover, 2, v, and 6 depend only on f and not on the choice of B. 
12. (Sylvester’s law of inertia for Hermitian matrices, cf. 12.2) If A € C"*" is a Hermitian matrix, then 
A is *congruent to the diagonal matrix D = I, © —1, © 05, where (z, v,5) = in(A). 
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13. There are exactly two invariants of n x n Hermitian matrices under *congruence, namely the rank 
and the signature. 

14. Two Hermitian n x n matrices are *congruent if and only if they have the same rank and the same 
signature. 

15. The signature of a Hermitian form is well-defined. 

16. Two Hermitian forms are “equivalent if and only if they have the same rank and the same signature. 

17. [HJ91, Theorem 1.3.5] Let A, B € C”*” be Hermitian matrices. Suppose that x € C", x* Ax = 
x*Bx =0 => x= 0. Thendi a,b € R such that aA + bB is positive definite. This fact can be 
obtained from [HJ91], where it is stated in a slightly different form, using the decomposition of 
every square, complex matrix as a sum of a Hermitian matrix and a skew-Hermitian matrix. 

18. The group of linear operators preserving the Hermitian form f(u,v) = >¢/_, u;¥; on C" is the 
n-dimensional unitary group. 


Examples: 


1. Let A € C"*" be a Hermitian matrix. The map f:C” x C” — C defined by f(u,v) = 
Die Leja 4ijui¥; is a Hermitian form on C". 

2. Let wy, Wa,... , We belinear functionals on V,andlet a), a2,...,a, € R.Thenthemap f: Vx V>C 
defined by f(u,v) = ar a;W;(u);(v) is a Hermitian form on V. 

3. Let H € C”*" bea Hermitian matrix. 
The map f :C"*" x C"*" — C defined by f(A, B) = tr(AH B*) is a Hermitian form. 
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13.1 Multilinear Maps 


Unless otherwise stated, within this section V, U, and W as well as these letters with subscripts, superscripts, 
or accents, are finite dimensional vector spaces over a field F of characteristic zero. 


Definitions: 


A map g from V,; x--- x V,, into U is a multilinear map (m-linear map) if it is linear on each coordinate, 
ie., for every v;,v; € Vj, i = 1,...,mand for everya € F the following conditions hold: 


(a) y(v},. ee Vi +vi,. . -> Vm) = (vy. sey Vio-- Vin) + v(v,. Leo Vise . ves 
(b) O(v1,...,4Vj,-.-5 Vin) = AQ(V1,...5Vin. +5 Vm)- 


The 2-linear maps and 3-linear maps are also called bilinear and trilinear maps, respectively. 

If U = F then a multilinear map into U is called a multilinear form. 

The set of multilinear maps from V| x -- - x V,, into U, together with the operations defined as follows, 
is denoted L(Vi,..., Vin3 U). For m-linear maps g, y, anda € F, 


(y oh g)(1;. on Vm) = w(v4,. oe Vm) + g(v1, tee >Vm)s 
(a~)(V1,.--5V¥m) = aQ(Y1,.--5 Vm). 


Let (bj1,...,bi,,) be an ordered basis of V;,i = 1,...,m. The set of sequences (j),..-; jm), 1 < ji < 
nj,i = 1,...,m, will be identified with the set D(11,...,,,) of maps a from {1,...,m} into N satisfying 
1 <a(i) < nj,i =1,...,m. 

For a € T'(nj,...,M%m), the m-tuple of basis vectors (bye(1))---» Dina(m)) is denoted by by. 
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Unless otherwise stated ['(1,,..., 1) is considered ordered by the lexicographic order. When there is 
no risk of confusion, I’ is used instead of [(n,,..., 1m). 

Let p,q be positive integers. If p is an (p + q)-linear map from W, x --- x W, x V, x --- x V, into 
U, then for each choice of w; in W;,i = 1,..., p, the map 


(V1,.--5Vq) FH O(W1,.. > Wp Vis +++ Vq)) 
Pw, ‘yet! Ww (Vio Vq) = P(W1,..-> Wp Vio-- +5 Vq)- 


Let n bealinear map from U into U’ and 6; alinear map from V/ into Vj,i = 1,...,m.If(W1,...5Vm) 
9(V),...,Vm) isa multilinear map from V; x --- x V,, into U, L(61,...;6mn37)(@) denotes the map from 
from V/ x --- x V), into U’, defined by 


(Vi5-2 29 V i) EH 1(P(OL(V4)5--- > Om(V),)))- 


Facts: 
The following facts can be found in [Mar73, Chap. 1] and in [Mer97, Chap. 5]. 


1. If g is a multilinear map, then g(vj,...,0,...,Vm) = 0. 
2. The set L(Vi,..., Vin3 U) is a vector space over F. 
3. If gy is an m—linear map from V, x --- x V,, into U, then for every integer p, 1 < p < m, and 


v; € Vj, 1 <i < p, the map Pvj,...vp 18 an (m — p)-linear map. 

4. Under the same assumptions than in (3.) the map (v),..., Vp) t= Pvt yeop from V, x --- x V, into 
L(Vp41,.-+» Vin3 U), is p-linear. A linear isomorphism from L(V,..., Vp, Vp4is.-+> Vin3 U) into 
L(V,,...) Vps L(Vp4i,--+» Vini U)) arises through this construction. 

5. Let 7 bea linear map from U into U’ and 6; a linear map from V/ into V;,i = 1,...,m. The map 
L(61,...,0m3) from L(Vj,...; Vin3 U) into L(V/,..., V/; U’) is a linear map. When m = 1, and 
U = U' = F, then L(6j, I) is the dual or adjoint linear map 6 from V;* into V,*. 

6. | (m1,...5%m)| = TfL, ni where | | denotes cardinality. 

7. Let (Yw)aer be a family of vectors of U. Then, there exists a unique m-linear map g from 
Vi x +--+ x Vp into U satisfying y(by) = ya, for everya eT. 

8. If (uj,...,U,) is a basis of U, then (9,4 : a € Ti = 1,...,m) is a basis of L(Vi,...; Vin3 U), 
where ¢j,q is characterized by the conditions ¢,4(bg) = 6c,gu;- Moreover, if g is an m-linear map 
from V, x --- x V,, into U such that for eacha € I, 


g(ba) = S- Qj,q Uj 


i=1 


then 
Q= Ss GiaPi,a- 
Qt 


Examples: 

The map from F” into F, (41,...,4m) — []j2, ai, is an m-linear map. 

Let V be a vector space over F. The map (a,v) +> av from F x V into V isa bilinear map. 

The map from F” x F™ into F, ((a),..-54m),(b1,.--50m)) HR >> a;bj, is bilinear. 

Let U, V, and W be vector spaces over F. The map (0,7) +» 6n from L(V, W) x L(U, V) into 

L(U, W), given by composition, is bilinear. 

5. The multiplication of matrices, (A, B) +> AB, from F”*" x F"*? into F””?, is bilinear. Observe 
that this example is the matrix counterpart of the previous one. 


ee 
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6. Let V and W be vector spaces over F . The evaluation map, from L(V, W) x V into W, 
(0,v) +> O(v), 


is bilinear. 
7. The map 


(aus A21>--- »Am1)) sree (41m A2m>++- >4mm)) =z det([a;;]) 


from the Cartesian product of m copies of F™ into F is m-linear. 


13.2 Tensor Products 


Definitions: 
Let Vi,..., Vin. P be vector spaces over F. Let v : Vi x --+ X Vin H-> P bea multilinear map. The pair 
(v, P) is called a tensor product of Vi,..., Vj, or P is said to be a tensor product of V,,..., Vi. with 


tensor multiplication v, if the following condition is satisfied: 


Universal factorization property 
If y is a multilinear map from V, x --- x V,, into the vector space U, then there exists a unique 
linear map, h, from P into U, that makes the following diagram commutative: 


Vi SQae se5°3G Vin eae P 
h 
9 
U 
Le, hv = 9. 
If P isa tensor product of Vj,..., Vin, with tensor multiplication v, then P is denoted by Vi @---@ Vin 
and v(vj,...,Vm) is denoted by v; ® --- ®@ v,, and is called the tensor product of the vectors v), .. . Vin. 


The elements of V; @ --- ® V,, are called tensors. The tensors that are the tensor product of m vectors 
are called decomposable tensors. 

When V, = --- = Vin = V, the vector space V; @ --- ® Vi, is called the mth tensor power of V and 
is denoted by Q” V. It is convenient to define @° V = F and assume that 1 is the unique decomposable 
tensor of @° V. When we consider simultaneously different models of tensor product, sometimes we use 
alternative forms to denote the tensor multiplication like @’, ®, or @ to emphasize these different choices. 

Within this section, V;,..., Vi, are finite dimensional vector spaces over F and (bj),..., bj, ) denotes 
a basis of V;,i = 1,...,m. When V is a vector space and x;,...,x, € V, Span({x,,...,x,}) denotes the 
subspace of V spanned by these vectors. 


Facts: 
The following facts can be found in [Mar73, Chap. 1] and in [Mer97, Chap. 5]. 


1. If Vi ®---® V,, and V, @’--- @’ V,, are two tensor products of V;,..., Vin, then the unique linear 
map h from V; ®@ --- ® Vi, into V; @’--- @’ V,, satisfying 


A(v, @ +++ @ Vm) = Vi B! +++ @! Vin 


is an isomorphism. 


13-4 


10. 
11. 


12. 


Handbook of Linear Algebra 


If (v(by) wer(n,...,4m) 18 a basis of P, then the pair (v, P) is a tensor product of Vj,..., Vin. This 
is often the most effective way to identify a model for the tensor product of vector spaces. It also 
implies the existence of a tensor product. 


. If P is the tensor product of Vi,..., Vi, with tensor multiplication v, and h : P +> Q isa linear 


isomorphism, then (hv, Q) is a tensor product of Vj,...,; Vin- 

When m = 1, it makes sense to speak of a tensor product of one vector space V and V itself is used 
as a model for that tensor product with the identity as tensor multiplication, i.e., @! V=V. 
Bilinear version of the universal property — Given a multilinear map from V; x --- x Vk x U; x 
-++ x U,, into W, 


(v1, +++) Vk, U),... ,Umn) Ee y(v1, +++) Vk, U],... ,Um)> 
there exists a unique bilinear map x from (V, ®---@ Vz) x (U; ®--- @ U,,) into W satisfying 


X(V} @--- OV, Uy @--- @u,,) = O(Vy,..., VE, W,..-,Um)s 
vi € Vi uj € Uj, i= 1,...,k, f= 1,...,m. 


Leta € F and vj,v; € V;, i = 1,...,m. As the consequence of the multilinearity of ®, the 
following equalities hold: 


(a) V1 @-+- @ (Vit Vj) @--- @Vn 
=V1@+*@Vji @---@Vm + Vi @--- OV; @+-- @Vm, 

(b) a(v) @ +++ @Vin) = (€V1) @ ++ @Vmn = ++ =V1 @ +++ @ (AVn), 

(c) Vi @-+-@0@---@Vm = 0. 

If one of the vector spaces V; is zero, then V; ® --- @ Vin, = {0}. 

Write b® to mean 


b® = bie) ®::-@ Dinain)- 
Then 
(bY wer 


is a basis of V; @- -- @ V,,. This basis is said to be induced by the bases (bi,..., bin; ), 7 = 1,...,m. 
The decomposable tensors span the tensor product V; ®--- ® Vi. Furthermore, if the set C; spans 
V;, i =1,...,m, then the set {v) @--- @ Vm : vj € Cj, i=1,...,m} spans Vj @--- @ Vin. 
dim(V, ® +++ ® Vn) = F[%, dim(V)). 

The tensor product is commutative, 


Vi @V2.= V2 @ Vis 


meaning that if V; ® V) is a tensor product of V; and V2, then V; ® V) is also a tensor product of 
V, and V, with tensor multiplication (v2, Vv) H Vv) @ Vo. 
In general, with a similar meaning, for anyo € S,,, 


Vi @::-@Vn= o(1) @ +++ @ Voun)- 
The tensor product is associative, 
(Vi @ V2) @ V3 = Vi @ (V2 @ V3) = Vi @ V2 @ V3, 


meaning that: 
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(a) A tensor product V; ® V2 ® V3 is also a tensor product of Vi ® V2 and V3 (respectively of V; and 
V, ® V3) with tensor multiplication defined (uniquely by Fact 5 above) for v; € V;,i = 1,2, 3, 
by (vi ® v2) @ v3 = Vi ® V2 ® vs (respectively by v; ® (v2 ® v3) = vi ® V2 @ Vs). 


(b) And, V; ® V2) @ V3 (respectively V; ® (V2 ®@ V3) is a tensor product of Vi, V2, V3 with tensor 
multiplication defined by v; ® v2 ® v3 = (Vv; ® v2) ® v3,v; € Vi, i = 1,2,3 (respectively 
vi ® v2 ® v3 =Vi ® (v2 ® v3), Vi € Vi; 1 —_ 1;2,3); 


In general, with an analogous meaning, 


(Vi @ ++» @ Vi) @ (Viti @ +++ @ Vin) =Vi @---@ Ving 


for anyk,1<k <m. 


Let W; be a subspace of V;, i = 1,...,m. Then W, ® --- ® W,, is a subspace of V; @ --- @ Vin, 
meaning that the subspace of V; ® --- © V;, spanned by the set of decomposable tensors of the 
form 


Wi © +--+ @ Win wi € Wj,i=1,...,m 
is a tensor product of W,,..., W, with tensor multiplication equal to the restriction of ® to 
W, xX -++ X Way. 
From now on, the model for the tensor product described above is assumed when dealing with 
the tensor product of subspaces of Vj. 


Let W,, W; be subspaces of V, and W, and W; be subspaces of V2. Then 


(a) (WM, @ Wi) (W; @ W;) = (WN Wy) @ (W212 WY). 
(b) Wi @ (W, + W,) = (MW, ® W2) + (MW, @ WY), 
(W, + W;) ® W. = (WM, @ Wa) + (W; @ W)). 


(c) Assuming W, N W; = {0}, 


(W, @ W,) @ Wa = (Wi @ Wa) @(W, @ W). 


Assuming W,M W; = {0}, 


W, @ (W2 © W5) = (W, @ Wr) 6 (WM, @ W). 


In a more general setting, if Wjj, j = 1,..., pj are subspaces of V;,i € {1,..., m}, then 


Pi Pm 
(5 vi) @-:-® (5 wi) = So Wy @-++® Wmyim- 


vel (pi pm) 


Pi Pm 
(om) @---® (6 vi) = QD Wy B+ @ Wnyim- 


VET (Piss Pm) 
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Examples: 


1. 


The vector space F”*" of the m x n matrices over F is a tensor product of F” and F" with 
tensor multiplication (the usual tensor multiplication for F’”*") defined, for (a),...,;dm) € F™ 
and (bj,...,b,) € F", by 


(@1,..-54m) ® (b1,...50n) = : b, ee by : 


Am 


With this definition, e; ® ej = E;; where e;, ej; and Ej; are standard basis vectors of F™, F", and 
F mxn . 


. The field F, viewed as a vector space over F,, isan mth tensor power of F with tensor multiplication 


defined by 
a, @-+-@ ay =] [ai, a; € F, i=l,...,m. 
i=l 
The vector space V is a tensor product of F and V with tensor multiplication defined by 
a@v=av, acF, ve’. 


Let U and V be vector spaces over F. Then L(V;U) is a tensor product U @ V* with tensor 
multiplication (the usual tensor multiplication for L(V; U)) defined by the equality (u@® f)(v) = 
f(vju, ue U,ve V. 
Let Vi,..., Vn be vector spaces over F. The vector space L(Vi,..., Vin3 U) is a tensor product 
L(V,,..-5 Vin3 F) ® U with tensor multiplication 

(9 @ u)(v},..-5 Vin) = G(V1,.--5Vn)U- 


Denote by F"'*""*"" the set ofall families with elements indexed in {1,...,m1} +--+ {1,...,Mm} = 
T'(m,...,%m). The set F"'*"*"" equipped with the sum and scalar product defined, for every 
(jis. ++ im) € T(n1,...;Mm), by the equalities 


(Bjpresin) FH (Birsesind = (Bityesin 7 Ojiscnim )? 


(Aj. sjim) = (CAj,.nind> «© F, 


is a vector space over F. This vector space is a tensor product of F",..., F"" with tensor multi- 
plication defined by 


(ay1,---541n,) ®:--@ (Ainty ++ +> Amny, ) = (I«s) 
i=l i 


The vector space L(Vi,..., Vin3 F) isa tensor product of V, = L(V; F),..., VX = L(Vin3 F) with 
tensor multiplication defined by 


g1 ®--:- @ &m(V15- P 23m) = [[g:@)- 
t=1 


Very often, for example in the context of geometry, the factors of the tensor product are vector 
space duals. In those situations, this is the model of tensor product implicitly assumed. 
The vector space 


L(V;...5 Vins F)* 
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is a tensor product of V},..., Vi, with tensor multiplication defined by 


vi ® ee @ Vin) = wv. . Vin): 


9. The vector space L(Vi,..., Vin3 F) isa tensor product L(V,,..., Vis F) @L(Vegi,---> Vin3 F) with 


tensor multiplication defined, for every v; € V;, i = 1,...,m, by the equalities 


(g ® Ww), 2 ge Vm) = g(V1,. 28 Vi) W (Vir ho Vn): 


13.3. Rank of a Tensor: Decomposable Tensors 


Definitions: 


Let z € V| ®--- ® V,,. The tensor z has rank k if z is the sum of k decomposable tensors but it cannot be 
written as sum of ] decomposable tensors, for any / less than k. 


Facts: 


The following facts can be found in [Bou89, Chap. I, §7.8]and [Mar73, Chap. 1]. 


1. The tensor z = v| © wi +--- + Vv; @w, € V @ W has rank ¢ if and only if (v),...,v,;) and 
(wi,...,W;) are linearly independent. 

2. Ifthe model for the tensor product of F™ and F" is the vector space of m x n matrices over F with 
the usual tensor multiplication, then the rank of a tensor is equal to the rank of the corresponding 
matrix. 

3. If the model for the tensor product U @ V* is the vector space L(V; U) with the usual tensor 
multiplication, then the rank of a tensor is equal to the rank of the corresponding linear map. 

4. x; @--- @X,, = 0 ifand only if x; = 0 for somei € {1,...,m}. 

5. Ifx;, yj are nonzero vectors of V;,i = 1,...,m, then 

Span({x; @ --- ®@ Xm}) = Span({y1 ® --- @ ym}) 
if and only if Span({x;}) = Span({y;}),7 = 1,...,m. 
Examples: 
1. Consider as a model of F” @ F", the vector space of the m x n matrices over F with the usual 


tensor multiplication. Let A be a tensor of F” @ F". Ifrank A = k (using the matrix definition of 
rank), then 


i 0 
A=M N, 
a 
where M = [x) - ++ X,] is an invertible matrix with columns x),..., xX» and 
yl 
y2 
N=|. 
Yn 
is an invertible matrix with rows y),...,¥n. (See Chapter 2.) Then 


A=xX, @yit-:- +X @ yz 


has rank k as a tensor . 
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13.4 Tensor Product of Linear Maps 


Definitions: 


Let 6; be a linear map from V; into Uj,i = 1,...,m. The unique linear map h from V; ® --- @ Vj, into 
U, ®@--- ® Uy, satisfying, for all vj € Vj,i = 1,...,m, 


h(v, @ +++ @Vn) = (v1) @ ++ @ On(Vn) 


is called the tensor product of 6),...,6,, and is denoted by 6; ® --- @ On. 

Let A; = (a\?) be an r; X s; matrix over F, t = 1,...,m. The Kronecker product of Aj,..., Am, 
denoted A; ®--- ® Ay, is the ([]/, r:) x ([]/2, s+) matrix whose (a, 8)-entry (a € T'(r,...,7m) and 
B €T(s1,..-55m)) is TT, ae ala: (See also Section 10.4.) 


Facts: 


The following facts can be found in [Mar73, Chap. 2] and in [Mer97, Chap. 5]. 
Let 6; be a linear map from V; into U;,i = 1,...,m. 


1. If; isa linear map from W,; into V;,i = 1,...,m, 
(61 @ +++ @ An)(M1 @ ++» @ Mm) = (01m) @ +++ @ (Onn). 


2. TIvV,@---@V,, = Ty, ®---@ Iy,,. 

3. Ker(0, ® --- ® Om) = Ker(0,) @ V2 ® --- @ Vin + Vi @ Ker(Qz) @---@ Vn +--- + Vi @---@ 
Vin—1 ® Ker(6,,). 
In particular, 6; ® --- @ O, is one to one if 6; is one to one, i = 1,...,m, [Bou89, Chap. II, $3.5]. 

4. 0, @ ++? @On(Vi @ +++ @ Vin) = O1(V) @ +++ @ On (Vin). In particular 6, ® --- @ O, is onto if 8; is 
onto,i = 1,...,m. 
In the next three facts, assume that 6; is a linear operator on the n;-dimensional vector space V;, 
i=l,...,m. 

5. tr(, @--- @ Om) = T]j, tr(;). 

6. Ifo (6;) = {aj1,...,4in,}, 1 = 1,...,m, then 


o(0, @---@On) = {Tam} 
i=l 


7. det(O; @ 6; ® +++ @ On) = det(O,)"2°°"™ det(@,)713°'"™ «- - det(O,,)7 2m! 

8. The map v : (0),...,4n) KH 0; @---@G,, isa multilinear map from L(V\; U;) x «++ L(Vin3 Um) 
into L(V, @ ++: @ Vins U; @ +++ @ Um). 

9. The vector space L(V, ® --- @ Vin; U; ® --- @ Um)) is a tensor product of the vector spaces 
L(Y; U,),...;L(Vin3 Um), with tensor multiplication (6),...,0m) KH 01 @ +++ @ On : 


L(Vi3 U1) @ +++ @ LVin3 Um) = L(V @ +++ @ Vins U1 @ +++ @ Um). 


10. As a consequence of (9.), choosing F as the model for @” F with the product in F as tensor 
multiplication, 


Vi @---@Vi=(Vi@---@Vn)*. 
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11. Let (Vij) j=1,...n, be an ordered basis of V; and (uj;) j=1,...q; an ordered basis of Uj,i = 1,...,m. 
Let A; be the matrix of 6; on the bases fixed in V; and U;. Then the matrix of 6; ® --- ® 6,, on 
the bases (V2 )yer(m,..sm,) ANd (UP )wer(q,,..09,) (induced by the bases (vjj) j=1,..4n, aNd (Wij) j=1,...95> 


respectively) is the Kronecker product of Aj,..., Ams 
12. Let m,...5M%ms11,--+5Tms ty ..-5tm be positive integers. Let A; be an n; x rj matrix, and B; be an 


r, X t; matrix, i = 1,...,m. Then the following holds: 


(a) (Ai @--+ @ Am)(Bi @ +++ @ Bm) = Ai Bi @--- @ AmBm: 
(b) (Al @ +++ @ Ak) @ (Ak+1 @ +++ @ Am) = Ar @ +++ @ Am. 


Examples: 
1. Consider as a model of U @ V*, the vector space L(V; U) with tensor multiplication defined by 


(u@ f)(v) = f(v)u. Use a similar model for the tensor product of U' and V™. Let 7 € L(U;U’) 
and 6 € L(V’; V). Then, for all € U @ V* = L(V;U), 


n @O*(E) = né0. 


2. Consider as a model of F” @ F", the vector space of the m x n matrices over F with the usual 
tensor multiplication. Use a similar model for the tensor product of F” and F*. Identify the set of 
column matrices, F”™*!, with F” and the set of row matrices, F!*", with F"”. Let Abe anr x m 
matrix over F. Let 64 be the linear map from F” into F" defined by 


ay 
a2 
Oa(ai,. oe Am) =A 


Am 


Let B beans x n matrix. Then, for all C € F”*" = F™ @ F", 0, @ Op(C) = ACB". 


3, For every i = 1,...,m consider the ordered basis (bj,...,bjn;) fixed in V;, and the basis 
(bj,,... »bj,.) fixed in Uj. Let 6; be a linear map from V; into Uj and let Aj = (ai?) be the s; x nj; 


matrix of 6; with respect to the bases (bj,,..., Din; ), (bi,,..- »bi. ). For every z € Vi @---@ Vas 


nm M2 Mm 


j=l p=l jm=1 


= po Cyb®. 


ael(n,...5m) 


Then, for B = (i,...,im) € T'(s1,---55m), the component ci, 
basis element bj; @--- @ by,;, of U1 @ --- @ Un is 


i of 0; ® --- @ 6,,(z) on the 


ny Nn 
ros = (1) (m) 
CB = Cit, = > yee > 1a Fins jm fis-sim 
j=l jn= 


us 3 
= Ss (Ios) Cy. 
An) 


yer (n,..., i=l 
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4. If A = [ajj] isan p x q matrix over F and B isanr x s matrix over F, then the Kronecker product 
of A and B is the matrix whose partition inr x s blocks is 


a\,B a\2B -e9 a1,B 

a,B anB see az, B 
A@®B= 

4piB ap2B +++ ApgB 


13.5 Symmetric and Antisymmetric Maps 


Recall that we are assuming F to be of characteristic zero and that all vector spaces are finite dimensional 
over F. In particular, V and U denote finite dimensional vector spaces over F . 


Definitions: 


Let m be a positive integer. When Vj = V. = --- = V;, = V L™(V;U) denotes the vector space of the 
multilinear maps L(Vi,...; Vin3 U). By convention L°(V;U) =U. 
An m-linear map y € L(V; U) is called antisymmetric or alternating if it satisfies 


W(Vo(1) see > Vo(m)) = sgn(o)w(vi, tee >Vin)s o€ Sin 


where sgn(o ) denotes the sign of the permutation o. 
Similarly, an m-linear map g € L(V; U) satisfying 


Q(Vo(1)> see »Vo(m)) = g(v, tee Vin) 


for all permutations o € S,, and for allv,,...,V,, in V is called symmetric. Let S”(V;U) and A”(V; U) 
denote the subsets of L(V; U) whose elements are respectively the symmetric and the antisymmetric 
m-linear maps. The elements of A (V; F) are called antisymmetric forms. The elements of S’”(V; F) 
are called symmetric forms. 

Let In, be the set of all maps from {1,...,m} into {1,...,}, Le, 


m times 


The subset of I’,,,, of the strictly increasing maps a (@(1) < --- < a@(m)) is denoted by Qy,,,. The subset 
of the increasing maps a € Ty, (a(1) <--- < a(m)) is denoted by Ginn. 

Let A = [a;j] bean m x n matrix over F. Leta € Tp, and B € Pq,,. Then A[a|] be the p x q-matrix 
over F whose (i, j)-entry is dq(j),6(;)» Le, 


Ale|B] = [aa(i),a(j)]- 


The mth-tuple (1, 2,..., 1m) is denoted by 1,,. If there is no risk of confusion ¢ is used instead of ¢,,. 
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Facts: 


1. 


If m > n, we have Qy,, = Y. The cardinality of I’, is, the cardinality of Qinn is oe and the 
cardinality of Ginn is ("*""'). 


. A™(V;U) and S”(V; U) are vector subspaces of L(V; U). 
. Let wy € L(V; U). The following conditions are equivalent: 


(a) w is an antisymmetric multilinear map. 


(b) For1 <i < j < mandforallw,...,v, € V;, 


w(vi,. ++ Vi-b Vjo Vit1>-- +» Vj-1> Vio Vj+l-- Vin) 


= —v(v,. + +9 Vi-l Vio Vid1>---> Vj-1 Vj» Vj+1-- +) Vin). 


(c) For 1 <i < mand forall v,...,v, € V; 


wiv. ++yVi+l Vio--- Vin) —_ —wW(v,. +o Vio Vitl--- Vii) 


. Let wy € L(V; U). The following conditions are equivalent: 


(a) w isa symmetric multilinear map. 


(b) For1 <i < j < mand forallw,...,vn € V, 


wv). + +9 Vi-1l Vj>Vitl>---+»Vj—-1) Vis Vj+1-- +) Vin) 


= w(vi,. --oVi-b Vio Vit1>--->Vj-lb Vj Vj+l-- Vin): 


(c) For 1 <i < mand forall vy,...,v, € V; 


wv, see Vi41) Vio--- Vin) = w(vi,. ++ Vio Vi4l-- Vn). 


. When we consider L(V; U) as the tensor product, L(V; F) ® U, with the tensor multiplication 


described in Example 5 in Section 13.2, we have 
A™(V;U) =A™(V;F)@®U and S"(V;U)=S"(V;F)@U. 


Polarization identity [Dol04] If y is a symmetric multilinear map, then for every m-tuple 
(v],...5Vm) of vectors of V, and for any vector w € V, the following identity holds: 


g(v,. : a = 


De ero mG (WH ELV FoF EmVny WH ELV ++ + EmVin)> 


E1Em 


~ 2mm! 


where ¢; € {—1,+]1},i = 1,...,m. 


Examples: 


1. 


The map 


((au1; 21+. ,Am1)>- : «9 (Gims Gam « oe >4mm)) aad det([a;;]) 


from the Cartesian product of m copies of F™ into F is m-linear and antisymmetric. 
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2. The map 


((i1s 21) - + «5 Ami1)) ++ +> (Aims ams + +>4mm)) > per([ai;]) 


from the Cartesian product of m copies of F™ into F is m-linear and symmetric. 


3, The map ((a1,...,4n),(b1,...,0n)) H+» (ajbj — bja;) from F" x F" into F"*" is bilinear anti- 
symmetric. 
4. Themap ((a@1,...,@n),(b1,...50n)) > (aibj+bja;) from F" x F" into F"*" is bilinear symmetric. 


5. The map x from V™ into A”(V; F )* defined by 


X15. Vind (W) = WW. Vin) Viy-+-5Vm © V, 


is an antisymmetric multilinear map. 
6. The map x from V” into S(V; F )* defined by 


X(V15---5 Vin) (W) = WV,-- 5 Vin)s Vibe Vin € Vis 


is asymmetric multilinear map. 


13.6 Symmetric and Grassmann Tensors 


Definitions: 


Let o € S,, be a permutation of {1,...,m}. The unique linear map, from @" V into @'"V satisfying 
Vi @ +++ OV  Vo-101) @ +++ @ Vo-1(m)> Vi>--29Vm € V; 


is denoted P(o). 

Let w bea multilinear form of L(V; F) ando an element of S,,. The multilinear form (vj,...,Vin) 
W(Vo(1)>+++Vo(m)) is denoted wy. 

The linear operator Alt from @”V into ®”V defined by 


1 
Alt = —— )/ sen(o) P(o) 


* 6ESin 


is called the alternator. In order to emphasize the degree of the domain of Alt, Alt ,, is often used for the 
operator having ®)” V, as domain. 
Similarly, the linear operator Sym is defined as the following linear combination of the maps P (0): 


As before, Sym ,, is often written to mean the Sym operator having ®&)” V, as domain. 


The range of Alt is denoted by \\" V, i.e. A" V = Alt (@” V), and is called the Grassmann space of 
degree m associated with V or the mth-exterior power of V. 

The range of Sym is denoted by //” V, ie., VV" V = Sym (@” V), and is called the symmetric space 
of degree m associated with V or the mth symmetric power of V. 

By convention 


®'v A’v Viv F. 
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Assume m > 1. The elements of A” V that are the image under Alt of decomposable tensors of ®)” V 
are called decomposable elements of /\” V. If x),...,Xm € V,X1 A+++ A Xm denotes the decomposable 
element of /\” V, 


X, A+++ AX, = MAI (x; @ +--+ @X,), 


and x; A --+ A X» is called the exterior product of x),...,X. Similarly, the elements of \/” V that are 
the image under Sym of decomposable tensors of @)” V are called decomposable elements of \/” V. If 
X1,---5Xm € V,X, V+++ V Xp denotes the decomposable element of \/” V, 


X) Ve++ VX = m!Sym (x; @ --- @ Xn), 


and x; V --- V Xj is called the symmetric product of x),..., Xj. 
Let (b),...,b,,) be a basis of V. Ifa € Tn, b®, b?, and bY denote respectively the tensors 


b® = bai) ®---@ ba(m)s 
b? = bai) Art N beucm)> 
by = bai) Wotan bum): 


Let n and m be positive integers. An n-composition of m is a sequence 


fh = ([1,---5 Mn) 


of nonnegative integers that sum to m. Let Cy, be the set of n-compositions of m. 

Let A = (Aj,...,A,) be an n-composition of m. The integer A,!---A,! will be denoted by A!. 

Let a € Ty. The multiplicity composition of a is the n-tuple of the cardinalities of the fibers of a, 
(lja~!(1)|,..., a7! (a)|), and is denoted by Aq. 


Facts: 
The following facts can be found in [Mar73, Chap. 2], [Mer97, Chap. 5], and [Spi79, Chap. 7]. 


1. A™ Vand \/™ Vare vector subspaces of ®” V. 

2. The map o + P(o) from the symmetric group of degree m into L(@”"V;@™V) is an 
F -representation of S,,, Le., P(ot) = P(o)P(t) for any o,t € S,, and P(I) = Igmy 

3. Choosing L(V; F), with the usual tensor multiplication, as the model for the tensor power, 
®” V*, the linear operator P(o) acts on L™(V; F) by the following transformation 


(P(o)y) = Wo- 


4. The linear operators Alt and Sym are projections, i.e., Alt? = Alt and Sym? = Sym. 
5. Ifm = 1, we have 


Sym = Alt = Ig\ vy =Iy. 
6 A" V={zEe@"V : Pl(c)(z) =sgn(o)z, Vo € Sin}. 


V™V ={ze@"V : P(o)(z) =z, Vo € Sy}. 
8. Choosing L(V; F ) as the model for the tensor power @)” V* with the usual tensor multiplication, 


as 


A’ Vt=A"(V3F) and \f"V* = S"(V;F). 
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Q v=aA\'v=a\ vey. 
10. @°?V=A?V@ \’ V. Moreover for z € &’ V, 
z = Alt (z) + Sym (z). 
The corresponding equality is no more true in ®” V ifm # 2. 
ll. A” V = {0} ifm > dim(V). 
12. Ifm > 1, any element of (\” V is a sum of decomposable elements of /\” V. 
13. Ifm > 1, any element of \/” V is a sum of decomposable elements of \/” V. 


14. Alt (P(o)z) = sgn(o)Alt (z) and Sym (P(a)(z)) = Sym (z),z € @” V. 
15. The map A from V” into /\” V defined for v1,...,Vm € V by 


A(V136009Vm) =VW Arts AV 


is an antisymmetric m-linear map. 
16. The map v from V™ into \/" V defined for v1,...,Vm € V by 


V(V1,-009Vm) =V Vite V Vin 

is a symmetric m-linear map. 
17. (Universal property for /\"" V) Given an antisymmetric m-linear map y from V™ into U, there 
exists a unique linear map h from /\” V into U such that 
WV... Vin) = hiv A+++ AV), Vi>-++9 Vn € V, 
i.e., there exists a unique linear map h that makes the following diagram commutative: 
A 
yn ———_> /\my 


h 
y 


U 


18. (Universal property for \/” V) Given a symmetric m-linear map g from V™ into U, there exists a 
unique linear map h from \/” V into U such that 


Q(V1,--25Vn) =A Ve! V Vin) Vis-+29 Vim € Vs 


i.e., there exists a unique linear map h that makes the following diagram commutative: 


h 
9 
U 


Let p and q be positive integers. 
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19. 


20. 


21. 


22. 


23. 


24, 
25. 


26. 


27. 


(Universal property for ®)” V-bilinear version) If y is a (p + q)-linear map from V?T4 into 
U, then there exists a unique bilinear map x from ®@? V x @! V into U satisfying (recall Fact 5 
in Section 13.2) 


X(V1 @ +++ OVp,Vp41 @ +++ @Vpsq) = W(V1,.-- > Vp4q)- 


(Universal property for /\"" V-bilinear version) If y is a (p + q)-linear map from V?T4 into U 
antisymmetric in the first p variables and antisymmetric in the last q variables, then there exists a 
unique bilinear map x from A? V x /(\! V into U satisfying 


XV A+++ AV py Vp41 A+++ AVptq) = WW, «+ Vp tq): 


(Universal property for \/" V-bilinear version) If y is a (p + q)-linear map from V?T4 into U 
symmetric in the first p variables and symmetric in the last q variables, then there exists a unique 
bilinear map x from \/? V x \/7 V into U satisfying 


XV V+ V Vp Vp41 Vets V Vp4q) = OV. ++ Vp aq). 


If (bi,...,b,,) is a basis of V, then (be eer, is a basis of @”V, (be )weq,,, is a basis of A” V, and 
(bY )weG,,,, is a basis of \/” V. These bases are said to be induced by the basis (bj,...,b,). 
Assume L(V; F) asthe model for the tensor power of ®)” V*, with the usual tensor multiplication. 
Let (fi,...; fn) be the dual basis of the basis (b),...,b,,). Then: 


(a) For every gy € L™(V; F), 


9= D> olbe) f?. 


EP nn 
(b) For every g € A"(V, F), 


g= > vlba) fp. 


LE Qmn 


(c) For every y € S"(V, F), 


1 
g= > 11 Pw) fa 


EG mn 


dim @" V =n", dim" V = ("), and dim" V = ("*"). 
The family 


((uiby + +++ + Unby) V +++ V (Ubi + ++ + Madr) pein 
is a basis of \/” V [Mar73, Chap. 3]. 
Let X),...,X; be vectors of V and g),..., £m forms of V*. Let aj; = gi (xj), i,j = 1,...,m. Then, 


choosing (@)” V)* as the model for ®” V* with tensor multiplication as described in Fact 10 in 
Section 13.4, 


$1 @ +++ @ Yn(X A+++ AXm) = det[aj/]. 


Under the same conditions of the former fact, 


£1 @+** @ Ym(K V +++ V Xn) = per[aj;]. 
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28. Let (fi,..-5 fn) be the dual basis of the basis (b,,...,b,,). Then, choosing (@)” V)* as the model 


for)” V*: 
(a) 
Ga Vache 
is the dual basis of the basis (b® )yer,,, of @”V. 
(b) 


(C2) anv) 


is the dual basis of the basis (b}) aca, of \” V. 
(c) 


LE Qmn 


1 
ose ® 
(a (fi diy ‘) AEG mn 


is the dual basis of the basis (bY )vec,,, of V” V. 


mn 


Let vj,...,Vim be vectors of V and (b),...,b,,) be a basis of V. 


29. Let A = [ajj] be the n x m matrix over F such that vj = pa ajjb;, j =1,...,m. Then: 


(a) 
Vi S++ OV, = yy (Tl e.) b®; 
AED an t=1 
(b) 
Vi Ae AVn = ys) det A[a|e]b?; 
HE Qin 
(c) 
1 bY 
ViVi VV = S 5 Per Alele] an 
EG inn  O* 
30. v1 A--+ A Vm = Oifand only if (vj, ...,V,) is linearly dependent. 


31. v) V--+ V Vm = 0 ifand only if one of the vjs is equal to 0. 
32. Let uj,...,U,, be vectors of V. 


(a) If(vi,...,;Vm) and (u),...,U,,) are linearly independent families, then 
Span({u, A --- A up»}) = Span({v) A --- A Vin}) 
if and only if 
Span({uj,...,Un}) = Span({vj,...,Vn}). 
(b) If(vy,...,Vm) and (uj,...,U,,) are families of nonzero vectors of V, then 


Span({v, V --- V Vn }) = Span({u, V --- V uz }) 
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if and only if there exists a permutation o of S,,, satisfying 
Span({v;}) = Span({u,q}), i= 1,...,m. 
Examples: 
1. Ifm = 1, we have 
sym = Alt =Igiy=ly. 


2. Consider as a model of @? F", the vector space of the n x n matrices with the usual tensor multi- 
plication. Then /\* F” is the subspace of the n x n antisymmetric matrices over F and \/* F" is the 
subspace of the n x n symmetric matrices over F . Moreover, for (a1,...,@n),(b1,...50n) € F": 


(a) (a),..-54n) A (b1,..-50n) = [aibj — bj4;]j,j=1,...n- 
(b) (@1,...54n) V (b1,...50n) = [ajibj + bia; ]ijat,. sn 


With these definitions, e; Ae; = E;; — Ej; ande; ve; = Ej; + E 
standard basis vectors of F™, F", and F™*”. 
3. Forx Ee V,xV---Vx=m!x®---@x. 


ji» where e;, e;, and Ej; are 


13.7. The Tensor Multiplication, the Alt Multiplication, 
and the Sym Multiplication 


Next we will introduce “external multiplications” for tensor powers, Grassmann spaces, and symmetric 
spaces, Let p,q be positive integers. 


Definitions: 


The (p,q)-tensor multiplication is the unique bilinear map, (z,z’)  z@ z’ from (@? V) x (@! V) 
into @?*4 V, satisfying 


(v1 @ «++ @Vp) @ (Vp41 @ +++ @Vpeq) = Vi @ ++ @Vp4q- 


The (p,q)-alt multiplication (briefly alt multiplication ) is the unique bilinear map (recall Fact 20 in 
section 13.6), (z,z’) & z Az’ from (A? V) x (A4 V) into A?" V, satisfying 


(Vp A+++ AVp) A (p41 Ast? AVp4q) = Vi A+? AV pt: 


The (p,q)-sym multiplication (briefly sym multiplication ) is the unique bilinear map (recall Fact 21 
in section 13.6), (z,z’)  zV z’ from (V/? V) x (\V/4 V) into V?"4 V, satisfying 


(Vv) V +++ V Vp) V (Wp Viet? V Vp4q) = V1 Vio V Vp tg: 


These definitions can be extended to include the cases where either p or q is zero, taking as multiplication 
the scalar product. 

Let m,n be positive integers satisfying 1 < m < n. Leta € Qm». We denote by a the element of 
Qn—mn Whose range is the complement in {1,...,n} of the range of a and by @ the permutation of S,,: 


~_ 1 ois m m+1 -:: n 
C= kay oe aGny eth). ee ata) 
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Facts: 
The following facts can be found in [Mar73, Chap. 2], [Mer97, Chap. 5], and in [Spi79, Chap. 7]. 
1. The value of the alt multiplication for arbitrary elements z € /\? V and z’ € /\‘ V is given by 


(p+4q)! 
pq! 


ZAZ= Alt p4q(z® 2’). 


2. The product of z € \/? V and z’ € \/1 V by the sym multiplication is given by 


(p+4q) 


| 
zVzZ= Pia! ~SyM pig(z® Zi 


3. The alt-multiplication z A z’ and the sym-multiplication z V z’ are not, in general, decomposable 
elements of any Grassmann or symmetric space of degree 2. 

4. LetO #£z€ /(\" V. Then z is decomposable if and only if there exists a linearly independent family 
of vectors V),...,Vin Satisfying z A vj = 0,i = 1,...,m. 

5. If dim(V) =n, all elements of \\""' V are decomposable. 

6. The multiplications defined in this subection are associative. Therefore, 


P q r 
z@z@z",zE(%) V, zZze® V, zZ’e® V; 

P q r 
waw'raw'we /\ V, we V;, wel\ V; 


yvyvysve Vv, yeKf Vv, yreVv 


are meaningful as well as similar expressions with more than three factors. 
7. Ifwe A? V,w’ € A‘ V, then 


w Aw = (—-1)Plw aw’. 
8. Ifye VW? V,y’ € V1 V, then 
yvy=yvy'. 


Examples: 


1. When the vector space is the dual V* = L(V; F) ofa vector space and we choose as the models of 
tensor powers of V* the spaces of multilinear forms (with the usual tensor multiplication), then 
the image of the tensor multiplication 9 @ w (g € L?(V; F) andy € L1(V; F)) on(v,...,Vp4q) 
is given by the equality 


(y® W)(V1,.- 5 Vp4q) = O(V1,-- 25 Vp) W(Vp4is. + +> Vp4q): 


2. When the vector space is the dual V* = L(V; F) ofa vector space and we choose as the models for 
the tensor powers of V* the spaces of multilinear forms (with the usual tensor multiplication), the 
alt multiplication of g € A?(V;F) and w € A1(V; F) takes the form 


(PA W)M1,--->Vp+q) 
1 
= Sial sgn(o )P(Vo(1)>+-+»Vo(p))W(Vo(p4i)> +++» Vo(p+q))- 


TESp+q 


3. The equality in Example 2 has an alternative expression that can be seen as a “Laplace expansion” 
for antisymmetric forms 
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(PA WL --->Vp+q) 
= > sgn(@)P(Va(1)> +++ Valp)) W (Var (1)>+ ++» Var(q))« 


LEQd, p+q 


4. In the case p = 1, the equality in Example 3 has the form 


(PAW), --->Vq41) 
qtl : 
_ So(-)) Tew i)v, seo Vj-1 Vj4+l>--- »Vq+1): 


j=l 


5. When the vector space is the dual V* = L(V; F) of a vector space and we choose as the models 
of tensor powers of V* the spaces of multilinear forms (with the usual tensor multiplication), the 
value of sym multiplication of gp € S?(V; F) and y € S1(V; F) on (v,...,Vp+4q) is 


(QV WML, --->Vp+q) 
1 


~ 5g! 5 P(Vo(1)> +++» Vo(p))W (Vo (pti) +++» Vo(ptq)): 


TES p+q 


6. The equality in Example 5 has an alternative expression that can be seen as a “Laplace expansion” 
for symmetric forms 


(QV WV... Vp tq) 
=> De P(Ver(1)> «+ +9 Varp))W (Var (1)> ++ +> Vae(q))« 


LE Qp.p+q 
7. Inthe case p = 1, the equality in Example 6 has the form 
(QV WV, - +> Vq41) 


qtl 


= Ss; Q(v)WM1,...,Vj—1) Vjtio +++» Vq4i)- 
j=l 


13.8 Associated Maps 


Definitions: 


Let 0 € L(V;U). The linear map 0 @---@ 0 from @” V into &)” U (the tensor product of m copies of 
@) will be denoted by ®)” 6. The subspaces /\” V and \/” V are mapped by ®)” 6 into A” U and \/" U, 
respectively. The restriction of @)” 6 to /\" V and to \/” V will be respectively denoted, \" 6 e\/"" 0. 


Facts: 
The following facts can be found in [Mar73, Chap. 2]. 
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1. Let vj,...,Vm € V. The following properties hold: 

(a) A” O(v. A+++ A Vn) = O(v1) A+++ A O(Vin)- 

(b) VV" O(vy V «++ V Vin) = O(v1) V+ V O(Vn). 
2. Let@ € L(V;U) and n € L(W, V). The following equalities hold: 

(a) A” (6n) = A"(8) A" (n). 

(b) V"(6n) = V"(8) V"(n). 
3. A™(Iv) = Inn ys V" Uv) = Tym y- 
4, Let 6,n € L(V; U) and assume that rank (6) > m. Then 

N\e@=fA 1 

if and only if@ = an anda” = 1. 
5. Let 0,n € L(V;U). Then \/" 6 = \/" 7 if and only if 6 = an anda” = 1. 
6. If@ is one-to-one (respectively onto), then A\"" 6 and \/” @ are one-to-one (respectively onto). 


From now on @ is a linear operator on the n-dimensional vector space V. 
7. Considering /\” 6 as an operator in the one-dimensional space /\" V, 


(/\"0) ) = det(o)z, for all ze /\" V. 


8. If the characteristic polynomial of 6 is 


then 


ai =tr( \'6), = eg 


9. If O has spectrum o (0) = {Aj,...,An}, then 


det (/\"0) =der(oyr-), det (\"0) = dettay("="), 
Examples: 


1. Let A be the matrix of the linear operator 6 € L(V; V) in the basis (b;,..., b,,). The linear operator 
on /\" V whose matrix in the basis (b?)aeq,,,, is the mth compound of A is \\" 0. 


13.9 Tensor Algebras 


Definitions: 


Let A be an F -algebra and (Ax)xen a family of vector subspaces of A. The algebra A is graded by (Ax)ren 
if the following conditions are satisfied: 


(a) A= Qyen Ak: 
(b) A; Aj © Aj+; for every i, j €N. 
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The elements of A; are known as homogeneous of degree k, and the elements of U,,<j Ax are called 
homogeneous. 


By condition (a), every element of A can be written uniquely as a sum of (a finite number of nonzero) 
homogeneous elements, i-e., given u € A there exist uniquely determined uz € Ax, k € N satisfying 


u= >_> y. 


keN 


These elements are called homogeneous components of u. The summand of degree k in the former 
equation is denoted by [ul]k. 

From now on V isa finite dimensional vector space over F of dimension n. As before @* V denotes 
the kth-tensor power of V. 

Denote by &) V the external direct sum of the vector spaces @‘ V, k EN. Ifz; € ® V, z; is identified 
with the sequence z € ® V whose ith coordinate is z; and the remaining coordinates are 0. Therefore, 
after this identification, 


Qv-O@'v. 


keN 


Consider in ®@) V the multiplication (x, y) H x ® y defined for x, y € ® V by 


[Ix@yk= So [x] @lyl, kEN, 
rsEN 
r+s=k 


where [x], ®[y], is the (r, s )-tensor multiplication of [x], and [y], introduced in the definitions of Section 
13.7. The vector space ®) V equipped with this multiplication is called the tensor algebra on V. 

Denote by /\ V the external direct sum of the vector spaces A‘ V, k EN. Ifz € Ai V, z; is identified 
with the sequence z € /\ V whose ith coordinate is z; and the remaining coordinates are 0. Therefore, 
after this identification, 


Av-@A¥. 


Recall that ia V = {0} ifk > n. Then 
Le 
A= DIY 
k=0 
and the elements of /\ V can be uniquely written in the form 
ZotZtee +2, Zi a V, i=0,...,n. 


Consider in /A\ V the multiplication (x, y) & x A y defined, for x,y € A V, by 


[ZAG lOS* So ees lee “READ 5 os 
r,s €{0,... ,n} 
r+s=k 

where [x], A [y]; is the (r,s)-alt multiplication of [x], and [y], referred in definitions of Section 13.7. 
The vector space /\ V equipped with this multiplication is called the Grassmann algebra on V. 

Denote by \/ V the external direct sum of the vector spaces VEV, EEN. 

If z; € \V' V, we identify z; with the sequence z € \/ V whose ith coordinate is z; and the remaining 
coordinates are 0. Therefore, after this identification 


Vv=@\V'v. 


keN 
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Consider in \/ V the multiplication (x, y) x v y defined for x, y € \/ V by 


IxVylk= So [x] Vly, kEN, 
rsEN 
r+s=k 
where [x], V [y]; is the (7, s)-sym multiplication of [x], and [y], referred in definitions of Section 13.7. 
The vector space \/ V equipped with this multiplication is called the symmetric algebra on V. 


Facts: 
The following facts can be found in [Mar73, Chap. 3] and [Gre67, Chaps. II and II]. 


1. The vector space ®) V with the multiplication (x,y) t+ x ®@ y is an algebra over F graded by 
(@* V)ken» whose identity is the identity of F = @° V. 

2. The vector space A V with the multiplication (x,y) ty x A y is an algebra over F graded by 
( NM V)xcen whose identity is the identity of F = N° V. 

3. The vector space \/ V with the multiplication (x,y) H+ x Vv y is an algebra over F graded by 
(vk V)ken whose identity is the identity of F = \° V. 

4. The F-algebra ®@) V does not have zero divisors. 

5. Let B bean F -algebraand@ a linear map from V into B satisfying 0(x)0(y) = —0(y)0(x) for all x,y € 
V. Then there exists a unique algebra homomorphism h from /\ V into B satisfying h|V = 6. 

6. Let B bean F -algebraand@ alinear map from V into B satisfying 0(x)0(y) = 6(y)@(x), forall x,y € 
V. Then there exists a unique algebra homomorphism h from \/ V into B satisfying h|V = 6. 

7. Let (b;,...,b,) be a basis of V. The symmetric algebra \/” V is isomorphic to the algebra of 
polynomials in n indeterminates, F [x),...,x,], by the algebra isomorphism whose restriction to 
V is the linear map that maps b; into x;,i = 1,...,n. 


Examples: 


1. Let x),...,X, be n distinct indeterminates. Let V be the vector space of the formal linear com- 
binations with coefficients in F in the indeterminates x),...,x,. The tensor algebra on V is the 
algebra of the polynomials in the noncommuting indeterminates x),...,x, ([Coh03], [Jac64]). 
This algebra is denoted by 


F (Xtysces Xn) 
The elements of this algebra are of the form 


f(x. : Xn) = 5 Ss CauXa(1) @-:: ® Xa(m)> 


meEN CED nn 


with all but a finite number of the coefficients cy equal to zero. 


13.10 Tensor Product of Inner Product Spaces 


Unless otherwise stated, within this section V, U, and W, as well as these letters subscripts, superscripts, 
or accents, are finite dimensional vector spaces over R or over C, equipped with an inner product. 

The inner product of V is denoted by ( , )y. When there is no risk of confusion ( , ) is used instead. In 
this section F means either the field R or the field C. 


Definitions: 


Let 6 bea linear map from V into W. The notation 6* will be used for the adjoint of 6 (i.e., the linear map 
from W into V satisfying (0(x), y) = (x,0*(y)) for allx € V andy € W). 
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The unique inner product (, ) on V; ®--- ® V,, satisfying, for every vj,u; € Vj, i = 1,...,m, 
(V1 @ +++ @Ving ty @ +++ @ Um) = [vias vs 


is called induced inner product associated with the inner products (, )y,, i =1,...,m. 
For each v € V, let fy € V* be defined by f,(u) = (u,v). The inverse of the map v > f, is denoted 
by oy (briefly @). The inner product on V%*, defined by 


(fg) = (o(g),0(f))v; 


is called the dual of (, )y. 
Let U, V be inner product spaces over F . We consider defined in L(V; U) the Hilbert-Schmidt inner 
product, ie., the inner product defined, for 6,1 € L(V; U), by (0,7) = tr(n*@). 


From now on V; ® --- ® V;, is assumed to be equipped with the inner product induced by the inner 
products (, )y,, 7=1,...,m. 


Facts: 
The following facts can be found in [Mar73, Chap. 2]. 


1. The map v > fy is bijective-linear if F = R and conjugate-linear (i.e., cv > ¢ fy) if F = C. 

2. If (bj,,...,bjn,) is an orthonormal basis of V;,i = 1,...,m, then {b® : w €T'(m,...,nm)} isan 
orthonormal basis of V; ® --- ® Vin. 

3. Let 6; € L(Vj; W;), i = 1,...,m, with adjoint map 67 € L(W,, V;). Then, 


(0, @ +++ @ Om)" = OF @ +++ @ GO. 


4. If6; € L(Vj; V;) is Hermitian (normal, unitary),i = 1,...,m, then 0; @---@6,, is also Hermitian 
(normal, unitary). 

5. LetO € L(V; V). If ®”” 6 (\/” @) is normal, then @ is normal. 

6. Let @ € L(V; V). Assume that @ is a linear operator on V with rank greater than m. If A" 6 is 
normal, then @ is normal. 

7. Ifuy,...5Ums Vis---9Vm € V? 


(uy A+++ A Uns V1 A+++ A Vn) = m! det(uj, vj), 


(uy Vers V Un, V, V+ + V Vin) = mlper(uj, Vj). 


8. Let (b;,...,b,,) be an orthonormal basis of V. Then the basis (b® )ger,,,, is an orthonormal basis of 
@" VV, (4/4 b))acq,,, is an orthonormal basis of \" V, and (,/— by Jaca, isan orthonormal 


basis of /” V. 


Examples: 


The field F (recall that F = R or F = C) hasan inner product, (a, b) + (a,b) = ab. This inner product 
is called the standard inner product in F and it is the one assumed to equip F from now on. 


1. When we choose F as the mth tensor power of F with the field multiplication as the tensor 
multiplication, then the canonical inner product is the inner product induced in @” F by the 
canonical inner product. 

2. When we assume V as the tensor product of F and V with the tensor multiplication a ® v = av, 
the inner product induced by the canonical inner product of F and the inner product of V is the 
inner product of V. 
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3. Consider L(V; U) as the tensor product of U and V* by the tensor multiplication (u @ f)(v) = 
f(v)u. Assume in V* the inner product dual of the inner product of V. Then, if (v1,...,Vv,) is an 
orthonormal basis of V and 6,n € L(V; U), we have 


m 


= 5° (0(v;),n(vj)) = tr(n*9), 


j=l 


i.e., the associated inner product of L(V; U) is the Hilbert-Schmidt one. 

4. Consider F'*" as the tensor product of F” and F” by the tensor multiplication described in 
Example 1 in section 13.2. Then if we consider in F™ and F" the usual inner product we get in 
F™*" as the induced inner product, the inner product 


—T — 
(A,B) tr(B A)= i aijbij- 


5. Assume that in V,* is defined the inner product dual of ( , )y,,i = 1,...,m. Then choosing 
L(V\,..., Vn; F) as the tensor product of V,*,..., V,*, with the usual tensor multiplication, the 
inner product of L(Vi,...; Vin; F) induced by the duals of inner products on V;*, i = 1,...,mis 
given by the equalities 


= ye 9(bie(1)5 tee Dineen) ¥ (Die()> tee > Dinee(m))- 


acer 


13.11 Orientation and Hodge Star Operator 


In this section, we assume that all vector spaces are real finite dimensional inner product spaces. 


Definitions: 


Let V be a one-dimensional vector space. The equivalence classes of the equivalence relation ~, defined 
by the condition v ~ v’ if there exists a positive real number a > 0 such that v’ = av, partitions the set 
of nonzero vectors of V into two subsets. 

Each one of these subsets is known as an open half-line. 

An orientation of V is a choice of one of these subsets. The fixed open half-line is called the positive 
half-line and its vectors are known as positive. The other open half-line of V is called negative half-line, 
and its vectors are also called negative. 

The field R, regarded as one-dimensional vector space, has a “natural” orientation that corresponds to 
choose as positive half-line the set of positive numbers. 

If V is an n-dimensional vector space, /\" V is a one-dimensional vector space (recall Fact 22 in section 
13.6). An orientation of V is an orientation of /\" V. 

A basis (b),...,b,,) of V is said to be positively oriented if b; \ --- A b, is positive and negatively 
oriented if b; A --- Ab, is negative. 

Throughout this section /\" V will be equipped with the inner product ( , ),, a positive multiple of 
the induced the inner product, defined by 


where the inner product on the right-hand side of the former equality is the inner product of ®” V 
induced by the inner product of V. This is also the inner product that is considered whenever the norm 
of antisymmetric tensors is referred. 
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The positive tensor ofnorm 1 of \” V, uy, is called fundamental tensor of V or element of volume of V. 

Let V be a real oriented inner product space . Let 0 < m <n. 

The Hodge star operator is the linear operator %,,, (denoted also by %) from /\” V into A\""”" V 
defined by the following condition: 


(km(w),w) uy =wAw’, forall w’ € \" V. 


Let n > 1 and let V be an n-dimensional oriented inner product space over R. The external product 
on V is the map 


(V1. -65Vn—1) Fe Vp Xo X Vy = Wen (V1 A+ A Vn-1)s 


from V"~! into V. 


Facts: 


The following facts can be found in [Mar75, Chap. 1] and [Sch75, Chap. 1]. 


ul 


,b,,) is a positively oriented orthonormal basis of V, then uy = by A--- A by. 
b 


. If (bj,...,b,) is a positively oriented orthonormal basis of V, then 


I mb) — sgn(@)b’., ae Qian 


where @ and a are defined in Section 13.7. 


. Let (v,..., Vn) and (uy,...,u,) be two bases of V andv; = Svajjuj, j = 1,...,n.Let A= [aj;]. 


Since (recall Fact 29 in Section 13.6) 
Vi A+++ AV, = det(A)uy A+++ A Uy, 


two bases have the same orientation if and only if their transition matrix has a positive determinant. 


. %& is an isometric isomorphism. 
. %o is the linear isomorphism that maps 1 € R onto the fundamental tensor. 
. Kn Kim —_ (= 1) Ty om Vit 


Let V be an n-dimensional oriented inner product space over R. 
If m # 0 and m # n, the Hodge star operator maps the set of decomposable elements of A” V 


onto the set of decomposable elements of \\""” V. 
Let (x1,...,Xm) be a linearly independent family of vectors of V. Then 


Yi Av* AYn—m = kin (Xi A+++ AXm) 
if and only if the following three conditions hold: 
(a) Yi> +++ Yn—m € Span({x1, see Xm})+3 
(b) lly A+++ A Yn—mll = [x1 A +++ A Xml 


(c) (X1,---5Xims Yio--+>Yn—m) is a positively oriented basis of V. 
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9. If(v),...;V,—1) is linearly independent, v, x --- x v,—1 is completely characterized by the following 
three conditions: 


(a) vy X +++ X Vy-1 € Span({v},...,Vn—1})+. 
(b) |lvi x +++ X Va-al] = Ilvi A+++ A Vn-ill. 
(c) (Vj,..-5Vn—1V1 X +++ X Vy—1) is a positively oriented basis of V. 


10. Assume V* = L(V; F), with dim(V) > 1, is equipped with the dual inner product. Consider 
L(V; F) as a model for the mth tensor power of V* with the usual tensor multiplication. Then 
A" V* = A™(V; F). If A is an antisymmetric form in A”(V; F), then ¥,,(A) is the form whose 
value in (v1, ... ,V,—m) isthe component in the fundamental tensor ofA \Q~!(v,)A+--AQ7 1 (Vn—m)s 
where @ is defined in the definition of section 13.10. 


KA, see »Vn—m Uys =AA eo '(v1) ANC ODS Q'(Vn—m): 


11. Assuming the above setting for the Hodge star operator, the external product of vj,...,V,—1 is the 
image by o of the form (uy*)y,,...v,_, (recall that (uy=)y,,.wv,-) Wn) = Uve(V1,-- +5 Vn—1 Vin) Les, 


Vi xX te X Vy = O( (Uys )vy,..yvya): 


The preceeding formula can be unfolded by stating that for each v € V, (v,v) X +++ X Va-1) = 
uy* (vi, tee ,Vn—15V)- 


Examples: 


1. If V has dimension 0, the isomorphism 4 from A\° V = R into A\° V = R is either the identity 
(in the case we choose the natural orientation of V) or —I (in the case we fix the nonnatural 
orientation of V). 

2. When V has dimension 2, the isomorphism % is usually denoted by J. It has the property J 2—_] 
and corresponds to the positively oriented rotation of 2/2. 

3. Assume that V has dimension 2. Then the external product is the isomorphism J. 

4. If dim(V) = 3, the external product is the well-known cross product. 
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In this chapter, we have collected classical equalities and inequalities regarding the eigenvalues, the singular 
values, the determinant, and the dimensions of the fundamental subspaces of a matrix. Also included is a 
section on identities for matrix inverses. The majority of these results can be found in comprehensive books 
on linear algebra and matrix theory, although some are of specialized nature. The reader is encouraged to 
consult, e.g., [HJ85], [HJ91], [MM92], or [Mey00] for details, proofs, and further bibliography. 


14.1 Eigenvalue Equalities and Inequalities 


The majority of the facts in this section concern general matrices; however, some classical and frequently 
used results on eigenvalues of Hermitian and positive definite matrices are also included. For the latter, 
see also Chapter 8 and [HJ85, Chap. 4]. Many of the definitions and some of the facts in this section are 
also given in Section 4.3. 


Facts: 


1. [HJ85, Chap. 1] Let A € F"*", where F = C or any algebraically closed field. Let pa(x) = 
det(x I — A) be the characteristic polynomial of A, and A), A2,...,A, be the eigenvalues of A. Denote 
by Sp(Aq,..-5An)(k = 1,2,...,m) the kth elementary symmetric function of the eigenvalues (here 
abbreviated S;(A)), and by S;(A) the sum of all k x k principal minors of A. Then 


* The characteristic polynomial satisfies 


Pa(x) = (x — Ay) (x — Az) +++ (x — An) 
=x" — Si(A)x" | + Sy(A)x™? He (HD Sy (Ax + (HD Sn(A) 
=x" — $,(A)x"! + So(A)xt? ++ + (-1)" 1S, 1% + (—1)" Sn (A). 
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° SK (A) = Sea). : gia) = Sx (A)(k = 1,2,.. .) fl). 
* A= S)\(A) = Oj, aii = Ai and) = det A= S,,(A) = []j_, Ai. 


i=1 


. [HJ85, (1.2.13)] Let A() be obtained from A € C"*” by deleting row and column i. Then 


d n 
GPa) = » Pati) (x). 


Facts 3 to 9 are collected, together with historical commentary, proofs, and further references, in 
[MM92, Chap. III]. 


. (Hirsch and Bendixson) Let A = [aj] € C"*" and d be an eigenvalue of A. Denote B = [bi] = 


(A+ A*)/2 and C = [cj] = (A — A*)/(2i). Then the following inequalities hold: 


|A| < nmax |aijjl, 
isf 

|ReA| < n max |bjjl, 
i,j 


|ImA| < m max |cjl. 
1] 


Moreover, if A+ A? € R”™", then 


n(n — 1) 
|ImA| < max |cjj|,/ —~—. 
i,j 2 


. (Pick’s inequality) Let A = [aj] € R”*" and A be an eigenvalue of A. Denote C = [cj] = 


(A — A’) /2. Then 


IW 
|ImA| < max |cjj| cot | — }. 
ij 2n 


. Let A = [aj] € C"*” and A be an eigenvalue of A. Denote B = [bj] = (A + A*)/2 and 


C = [cy] = (A — A*)/(2i). Then the following inequalities hold: 


min{u : uw € o(B)} < Red < max{w: wu € o(B)}, 


} 
min{v : v € o(C)} <ImA < max{v: v € a (C)}. 


(Schur’s inequality) Let A = [aj] € C"*” have eigenvalues 4; (j = 1,2,...,n). Then 
Fees! i,j=l 


with equality holding if and only if A is a normal matrix (i.e., A*A = AA*). (See Section 7.2 for 
more information on normal matrices.) 

(Browne’s Theorem) Let A = [aj] € c"*" and Aj; = 1,2,...,2) be the eigenvalues of A ordered 
so that |A,| > |A2| > --- = |An|. Let also 0; > 02 > --- > oy be the singular values of A, which 
are real and nonnegative. (See Section 5.6 for the definition.) Then 


On <|Ajl<o (fj =1,2,...,n). 


In fact, the following more general statement holds: 
k k k 
[Joint < [a1 < [La 
i=l j=l i=l 


for every k € {1,2,...,n} and every k-tuple (t), h,...,%) of strictly increasing elements chosen 
from {1,2,...,n}. 
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8. Let A € C”*” and R;, C; (i = 1,2,...,n) denote the sums of the absolute values of the entries of 
A in row i and column i, respectively. Also denote 


R=max{R;} and C= max{C;}. 
Let A be an eigenvalue of A. Then the following inequalities hold: 


|A| Dee as Ba < Roy 
~~ 2 = 2 
|A| < max; ./RiC; < VRC, 
|A| < min{R, C}. 


9. (Schneider’s Theorem) Let A = [aj] € C"*” and Aj(j = 1,2,...,m) be the eigenvalues of A 
ordered so that |Aj| > |A2| => +--+ = |An|. Let x = [x;] be any vector in R” with positive entries and 
define the quantities 


Then 


for all n-tuples (i1,72,...,i,) of elements from {1,2,...,} such that 


Ni, 2 Zee 2 Ti,- 


1 


10. [HJ85, Theorem 8.1.18] For A = [aj] € C”*", let its entrywise absolute value be denoted by 
|A| = [|aj|]. Let B € C"*” and assume that | A] < B (entrywise). Then 


P(A) < p(lAl) < p(B). 
11. [HJ85, Chap. 5, Sec. 6] Let A € C”*" and || - || denote any matrix norm on C”*”. (See 
Chapter 37). Then 
p(A) < |All 
and 


lim | A‘|I!/* = p(A). 
k—0o 


12. [HJ91, Corollary 1.5.5] Let A = [aj] € C”””. The numerical range of A € C"*" is W(A) = 
{v*Av € C: v € C” with v*v = 1} and the numerical radius of A € C”*" isr(A) = max{|z| :z € 
W(A)}. (See Chapter 18 for more information about the numerical range and numerical radius.) 
Then the following inequalities hold: 


r(A™) < [r(A)]”) (m= 1,2,...), 


bis 4h + 1s 
WAN < (A) < [Allo 
A A\? 
Haze (ay SE Greet ae al). 


2 
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Moreover, the following statements are equivalent: 
(a) r(A) = ||All2. 

(b) p(A) = ||All2. 

(c) || A"ll2 = IAI. 

(d) JA‘ ll, =|]AllS (kK =1,2,...). 


Facts 13 to 15 below, along with proofs, can be found in [HJ85, Chap. 4]. 
13. (Rayleigh-Ritz) Let A € C”*” be Hermitian (i.e., A = A*) with eigenvalues A; > Az > +++ > An. 


Then 
(a) Anx*x < x* Ax <A,x*x forallx €C". 
* 
x* Ax 
(b) A, = max = max x* Ax. 
x40 x*x xx 
. x* Ax ye 
(c) A, = min = min x* Ax. 
x40 x*x x*x=1 
14. (Courant—Fischer) Let A € C"*" be Hermitian with eigenvalues 4; > Az > +--+ > An. Let 
k € {1,2,...,n}. Then 
: x* Ax 
AR = min max 
W1,W25.-5WE_-1 EC” x40,xeC" x*x 
X LW 5W 5----Wk-1 
x* Ax 
= max min 
W1,W2y.05Wn-KEC" x£0,xeC" x*x 
XLW)5W2)..9Wn—k 


15. (Weyl) Let A,B € C”*” be Hermitian. Consider the eigenvalues of A, B, and A + B, de- 
noted by A;(A),A;(B), A;(A + B), respectively, arranged in decreasing order. Then the following 
hold: 


(a) For eachk € {1,2,...,n}, 
Ak(A) + An(A) < ARCA + B) < ARCA) + Ai (B). 
(b) For every pair j,k € {1,2,...,n} such that; ++k>n+1, 
Ajtk-n(A + B) > Aj(A) + Ag(B). 
(c) For every pair j,k € {1,2,...,n} such that; +k<n+1, 


Aj (A) +Ag(B) = Aj4e-1(A + B). 


Examples: 


1. To illustrate several of the facts in this section, consider 


1 -l 0 2 
3 1 —2 1 

a 0 0 -1}’ 
= 2 1 0 


whose spectrum, o (A), consists of 
Ad, = —0.7112 + 2.67181, Ay = —0.7112 — 2.67187, A3 = 2.5506, A4 = 0.8719. 


Note that the eigenvalues are ordered decreasingly with respect to their moduli (absolute values): 


[Ai] = |Ag| = 2.7649 > |A3| = 2.5506 > |A4| = 0.8719. 
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The maximum and minimum eigenvalues of (A + A*)/2 are 2.8484 and —1.495. Note that, as 
required by Fact 5, for every A € o (A), 


—1.495 < |A| < 2.8484. 


To illustrate Fact 7, let (t), f) = (1,3) and compute the singular values of A: 


01 = 4.2418, 07 = 2.5334,03 = 1.9890, 04 = 0.7954. 


Then, indeed, 


0403 = 1.5821 < |A4||A3| = 7.0522 < ojo) = 10.7462. 


Referring to the notation in Fact 8, we have C = 6 and R = 7. The spectral radius of A is 
p(A) = 2.7649 and, thus, the modulus of every eigenvalue of A is indeed bounded above by the 
quantities 


REC 3 
" = 5 =65, JRC = 6.4807, min{R,C} =6. 


Letting B denote the entrywise absolute value of A, Facts 10 and 11 state that 


p(A) = 2.7649 < p(B) = 4.4005 and p(A) = 2.7649 < || All, = 4.2418. 


Examples related to Fact 12 and the numerical range are found in Chapter 18. See also Example 2 
that associates the numerical range with the location of the eigenvalues. 
2. Consider the matrix 


ooo 
oro 


and note that for every integer m > 2, A” consists of zero entries, except for its (1, 1) entry that is 
equal to 1. One may easily verify that 


P(A) = 1, |Alloo = [|All = IlAll2 = 1. 


By Fact 12, it follows that r(A) = 1 and all of the equivalent conditions (a) to (d) in that fact hold, 
despite A not being a normal matrix. 


14.2 Spectrum Localization 


This section presents results on classical inclusion regions for the eigenvalues of a matrix. The following 
facts, proofs, and details, as well as additional references, can be found in [MM92, Chap. III, Sec. 2], [HJ85, 
Chap. 6], and [Bru82]. 


Facts: 
1. (Gergorin) Let A = [aj] € C”*” and define the quantities 


n 
R= lagl) Gases 
j=l 
j#i 
Consider the Gersgorin discs (centered at a;; with radii R;), 


D; = {z€ C: |z—a;;| < Rj} Cia eee 0 
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Then all the eigenvalues of A lie in the union of the Gersgorin discs; that is, 
o(A) Cc |) Di. 
i=l 


Moreover, ifthe union of k Gersgorin discs, G, forms a connected region disjoint from the remaining 
n — k discs, then G contains exactly k eigenvalues of A (counting algebraic multiplicities). 
(Lévy—Desplanques) Let A = [a;] € C"*” bea strictly diagonally dominant matrix, namely, 


n 

ali laght se Site t a.3m): 
j=l 
jAi 


Then A is an invertible matrix. 
(Brauer) Let A = [aj] ¢ C””" and define the quantities 


RS Sy lag ASO: 
j=l 
j#i 
Consider the ovals of Cassini, which are defined by 
Vij ={zeC:|z—-a;j||z— ajjl < R Rj} (i, j = Lie hts j)- 


Then all the eigenvalues of A lie in the union of the ovals of Cassini; that is, 


o(A)C LJ Vij- 
j= 


[VK99, Eq. 3.1] Denoting the union of the GerSgorin discs of A € C”*" by (A) (see Fact 1) and 
the union of the ovals of Cassini of A by K (A) (see Fact 2), we have that 


o(A) C K(A) CT(A). 


That is, the ovals of Cassini provided at least as good a localization for the eigenvalues of A as do 
the Gersgorin discs. 
Let A = [aj] € C"*” such that 


n n 
laiillax| > D> lagl D> laxjl Gk = 1,2,...,n,1 #K). 
j=l j=l 
iti ik 
Then A is an invertible matrix. 
Facts 1 to 5 can also be stated in terms of column sums instead of row sums. 
(Ostrowski) Let A = [aj] € C"*" anda € [0, 1]. Define the quantities 


n n 
Rea ys lak Gs > lal (12,2 2259): 
j=l jet 
i#i i#i 
Then all the eigenvalues of A lie in the union of the discs 


D;(a) = {z €C:|z—a;;| < sy Oramce (i = 1,2,...,7)3 
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o(A) CJ Di(@). 
i=l 
8. Let A € C"*” and consider the spectrum of A, 0 (A), as well as its numerical range, W(A). Then 


a(A) Cc W(A). 


In particular, if A is a normal matrix (i.e., A*A = AA*), then W(A) is exactly equal to the convex 
hull of the eigenvalues of A. 


Examples: 
1. To illustrate Fact 1 (see also Facts 3 and 4) let 


31 1 05 —-1 0 

—-l 2 145 0 

A= 1 2 AT, 0 1 
0 -!l O 10 1 

1 0 1 -1 1 


and consider the Gerégorin discs of A displayed in Figure 14.1. Note that there are three connected 
regions of discs that are disjoint of each other. Each region contains as many eigenvalues (marked 
with +’s) as the number of discs it comprises. The ovals of Cassini are contained in the union of the 
GerSgorin discs. In general, although it is easy to verify whether a complex number belongs to an oval 
of Cassini or not, these ovals are generally difficult to draw. An interactive supplement to [VK99] 
(accessible at: www. emis.math.ca/EMIS/journals/ETNA/vol.8.1999/pp15-20. 
dir/gershini.htm1) allows one to draw and compare the GerSgorin discs and ovals of Cassini 
of 3 x 3 matrices. 
2. To illustrate Fact 8, consider the matrices 


1 -1l 2 24+ 21 2-i 1—2i 
A= 2 -1 O and B={14+2i -1l-i -—1-—2i 
—1 0 1 2+i —-2-i —l-i 
6- 
4b eed 
oe, ‘e 
ae \ 
a / xs 
/ 
A : \ 
| 
\ i 
/ 
abe j 
Se ue 
-4b Pa cl ae 
-6 | | | | | 
-10 -5 0 5 10 


FIGURE 14.1 The GerSgorin disks of A. 
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2.5 d 
9 | 1 
15 a8 
j 0.6 
Ly 04 
0.5 + 0.2 
0 0 
-0.5 | -0.2 
-0.4 
-1 | 0 
-0.6 
-1.5 + 08 
2 + -1 
2.5 L L L 1 1 L L L L L L L L 1 Ll Ll Ll 
-15 -1  -05 0 0.5 1 1.5 2 -1-0.8-0.6 -0.4-0.20 0.2 04 0608 1 


FIGURE 14.2 The numerical range of A and of the normal matrix B. 
Note that B is a normal matrix with spectrum {1,i,—1 — 1}. As indicated in Figure 14.2, the 


numerical ranges of A and B contain the eigenvalues of A and B, respectively, marked with +’s. 
The numerical range of B is indeed the convex hull of the eigenvalues. 


14.3 Inequalities for the Singular Values and the Eigenvalues 


The material in this section is a selection of classical inequalities about the singular values. Extensive details 
and proofs, as well as a host of additional results on singular values, can be found in [HJ91, Chap. 3]. 
Definitions of many of the terms in this section are given in Section 5.6, Chapter 17, and Chapter 45; 
additional facts and examples are also given there. 


Facts: 


1. Let Ae C”*” and o; be its largest singular value. Then 


0) = |[Allo- 
2. Let A € C”*", q = min{m, n}. Denote the singular values of A by 0) > 0) >--- > o, and let 
k € {1,2,...,q}. Then 
O,= min max || Ax|l2 
W1,W25.-5Wk-1 EC" \xllg=LxeC" 


XLW1,W25..5Wk-1 


= max min || Ax|l2 
W1,W25.05Wn-kKEC” \Ixllg=LxeC" 
XLW)5W25..Wn—k 


= min max || Ax||2 
wocn xeWw 
dim W=n—k+1 — ||x|]2=1 
= max min ||Ax|l2, 
wec xeW 
dimW=k —[[x|]2=1 


where the optimizations take place over all subspaces W C C” of the indicated dimensions. 
3. (Weyl) Let A € C”*" have singular values 0; > 02 > --- > 0, and eigenvalues A; (j = 1,2,...,n) 
be ordered so that |A,| > |A2| > --- > |An|. Then 


JAjAz +++ AR] S O10 +++ OK (k = 1,2,...,n). 


Equality holds in (3) when k = n. 
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4. (A. Horn) Let A € C”*? and B € C?*". Let also r = min{m, p},s = min{p,n}, and q = 


min{r,s}. Denote the singular values of A, B, and AB, respectively, by 0) > 02 > --- > o;, 
™%] 2% 2-+++2 t,and x1 > x2 2-++ 2 Xq- Then 
k k 
[Lx < [oz (KS 152) hu5 GQ): 
i=1 i=l 
Equality holds ifk = n = p = m. Also for any t > 0, 
k k 
Se = Gay (k = 1,2,...,q). 
i=l i=l 
5. Let A € C"*” have singular values 0, > 02 > --- > o, and eigenvalues Aj G = 1,2,...,1) 


ordered so that |A,| > |A2| >--- > |A,|. Then for any t > 0, 


k k 
Ses of GSLz aw) 
i=l i=l 


In particular, for t = 1 and k = n we obtain from the inequality above that 


n 
tA] < SoG. 
al 


6. Let A, B € C”*" and q = min{m, n}. Denote the singular values of A, B, and A+ B, respectively, 
byo, >. > 20,1 >) >: = ty, and Wh = w= --- = Wy. Then the following 


inequalities hold: 
(a) Wi4j-1<0 +7) USij<q, itj<qt)). 
(b) lpi Oj) ST (Gi = 1,2,...,q). 


k k k 
(©) Sows doa+ doy (k = 1,2,...,q). 
j=l j=l i=l 


7. Let A € C"*" have eigenvalues 4; (j = 1,2,...,1) ordered so that |Ai| > |A2| > --- 


Denote the singular values of Ak by o\(A*) > 05(A*) > --- > 0, (A*). Then 


lim [o;(A*)]/* = |A;| (i = 1,2,...,n). 
k—>oo 


Examples: 


1. To illustrate Facts 1, 3, and 5, as well as gauge the bounds they provide, let 


1 2 -1 0 

2 141 1 0 
a 21 1 1 0] 

0 1-1 1 0 


whose eigenvalues and singular values ordered as required in Fact 3 are, respectively, 


= |Anl. 


Ay = 2.6775 + 1.02271, 42 = —2.0773 + 1.46857, Az = 1.3998 — 0.49127, A, = 0, 


and 


01 = 3.5278, 07 = 2.5360,03 = 1.7673, 04 = 0. 
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According to Fact 1, || Al|2 = 0) = 3.5278. The following inequalities hold according to Fact 3: 


7.2914 = |AyA2| < a0) = 8.9465. 
10.8167 = |AyA2A3| < 10203 = 15.8114. 


Finally, applying Fact 5 with t = 3/2 and k = 2, we obtain the inequality 


2 


8.9099 = |A,|/? + [Ag |? < of? +037 = 10.6646. 


For t = 1 andk = n, we get 
4 
2.8284 = |2 + 2i| = |tr(A)| < 5 > oj = 7.8311. 
j=l 


14.4 Basic Determinantal Relations 


The purpose of this section is to review some basic equalities and inequalities regarding the determinant 
of a matrix. For most of the facts mentioned here, see [Mey00, Chap. 6] and [HJ85, Chap. 0]. Definitions 
of many of the terms in this section are given in Sections 4.1 and 4.2; additional facts and examples are 
given there as well. Note that this section concludes with a couple of classical determinantal inequalities 
for positive semidefinite matrices; see Section 8.4 or [HJ85, Chap. 7] for more on this subject. 

Following are some of the properties of determinants of n x n matrices, as well as classical formulas for 
the determinant of A and its submatrices. 


Facts: 
1. Let A € F"*". The following are basic facts about the determinant. (See also Chapter 4.1.) 

* det A= det A’; if F =C, then det A* = det A. 

¢ If B is obtained from A by multiplying one row (or column) by a scalar c, then det B = c det A. 

* det(cA) = c" det A for any scalar c. 

* det(AB) = det A det B. If A is invertible, then det A~! = (det A)7! 

* If B is obtained from A by adding nonzero multiples of one row (respectively, column) to other 
rows (respectively, columns), then det B = det A. 

* detA= oa aes, SSF )@10(1)420(2) * * * Ano (n)» Where the summation is taken over all permutations 
o of n letters, and where sgn(a) denotes the sign of the permutation o. 

* Let Aj denote the (n—1) x (n—1) matrix obtained from A € F"*" (n > 2) by deleting rowi and 
column j. The following formula is known as the Laplace expansion of det A along column j: 


detA=S°(-1)'ajdet Aj (j = 1,2,...,n). 


2. (Cauchy—Binet) Let A € F™k, B € Fx" and consider the matrix C = AB € F”*". Let also 
a C {1,2,...,m}and B C {1,2,...,n} have cardinality r, where 1 < r < min{m, k,n}. Then the 
submatrix of C whose rows are indexed by a and columns indexed by £ satisfies 


det C[a, B] = ys det A[a, y] det B[y, B]. 


YE(I,2,..4k} 
lyl=r 


3. [Mey00, Sec. 6.1, p. 471] Let A = [aj(x)] be ann x n matrix whose entries are complex differen- 
tiable functions of x. Let D; (i = 1,2,...,m) denote the n x n matrix obtained from A when the 
entries in its ith row are replaced by their derivatives with respect to x. Then 


i“ 
(det A) = = det Dj. 
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4. Let A = [aj] be ann x n matrix and consider its entries as independent variables. Then 


d(det A) 
dai 


= det A({i},{j}) (7 = 1,2,...,n), 


where A({i}, {j}) denotes the submatrix of A obtained from A by deleting row i and column j. 
5. [Mey00, Sec. 6.2] Let A € F"*" anda C ({1,2,...,n}. If the submatrix of A whose rows and 
columns are indexed by a, A[a], is invertible, then 


det A = det Ala] det(A/A[a]). 
In particular, if A is partitioned in blocks as 


Au A 


A= 
Az, Az2 


> 


where Aj; and A») are square matrices, then 


fey det Au det( Ar. = Ao (Aq1)7! Aq) if Ay, is invertible 
~ det An det(Aq, = Aj2(Ay2)7! Aa1) if An is invertible. 


The following two facts for F = C can be found in [Mey00, Sec. 6.2, pp. 475, 483] and [Mey00, 
Exer. 6.2.15, p. 485], respectively. The proofs are valid for arbitrary fields. 
6. Let A € F"*" be invertible and c,d € F”. Then 


det(A + cd’) = det(A)(1 + d?A7!c). 


7. Let A € F"*" be invertible, x, y € F”. Then 


det 


A x a T 
wi | = —det(A+ xy"). 


8. [HJ85, Theorem 7.8.1 and Corollary 7.8.2] (Hadamard’s inequalities) Let A = [aj] € C"*" bea 
positive semidefinite matrix. Then 


det A < [[ 4:- 


i=1 


If A is positive definite, equality holds if and only if A is a diagonal matrix. 
Fora general matrix B = [bj] ¢ C"*", applying the above inequality to B* B and B B*, respectively, 
one obtains 


If B is nonsingular, equalities hold, respectively, if and only if the rows or the columns of B are 
orthogonal. 
9. [HJ85, Theorem 7.8.3] (Fischer’s inequality) Consider a positive definite matrix 


xX Y 


BE Neg 


> 


partitioned so that X, Z are square and nonvacuous. Then 


det A < det X det Z. 
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Examples: 


For examples relating to Facts 1, 2, and 5, see Chapter 4. 


1. Let 
13 -l 2 
A=| 0 1 1] and x=]1]/,y=] 1 
—1 2 2 —l 
Then, as noted by Fact 7, 
3 -l 1 a aS 
Ax Oa 7 " 
det ges cal = lea , =-—det(A+xy’)=|]2 2 0} = 10. 
i Si 1 3 1 


Next, letting c = [121]? and d = [0 — 1 — 1]", by Fact 6, we have 
det(A + cd‘) = det(A)(1 +d? A~'c) = (—4) - (-1) = 4. 


2. To illustrate Facts 8 and 9, let 


3 1/1 
xX Y 
A=/1 5|/1)/=]). 5 
1 1)3 


Note that A is positive definite and so Hadamard’s inequality says that 
det A < 3-5-3 = 45; 
in fact, det A = 36. Fischer’s inequality gives a smaller upper bound for the determinant: 
det A < det X det Z = 13-3 = 39. 


3. Consider the matrix 


1 2 —2 
B= {4 -1 1 
0 1 1 


The first inequality about general matrices in Fact 8 applied to B gives 
| det B| < /9-18-2= 18. 


As the rows of B are mutually orthogonal, we have that | det B| = 18; in fact, det B = —18. 


14.5 Rank and Nullity Equalities and Inequalities 


Let A bea matrix over a field F . Here we present relations among the fundamental subspaces of A and their 
dimensions. As general references consult, e.g., [HJ85] and [Mey00, Sec. 4.2, 4.4, 4.5] (even though the 
matrices discussed there are complex, most of the proofs remain valid for any field). Additional material 
on rank and nullity can also be found in Section 2.4. 


Facts: 


1. Let Ae F™*". Then rank(A) = dimrangeA = dimrangeA’. 
If F = C, then rank(A) = dim range A* = dim range A. 
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. If Ae C”*", then rangeA = (ker A*)+ and range A* = (kerA)+. 
. If A € F™*" and rank(A) = k, then there exist X € F”** and Y € F**" such that A = XY. 
. Let A, B € F”*". Then rank(A) = rank(B) ifand only if there exist invertible matrices X € F’”*” 


and Y € F"*" such that B = XAY. 


. (Dimension Theorem) Let A € F”™*". Then 


rank(A) + null(A) =n and rank(A) + null(A’) = m. 


If F = C, then rank(A) + null(A*) = m. 


. Let A, B € F™*", Then 


rank(A) — rank(B) < rank(A + B) < rank(A) + rank(B). 


. Let Ae F™™", B € F™**, and C = [A|B] € F™™**), Then 


° rank(C) = rank(A) + rank(B) — dim(rangeA NM rangeB). 
¢ null(C) = null(A) + null(B) + dim(rangeA NM rangeB). 


. Let Ae F”*" and B € F"**, Then 


rank(AB) = rank(B) — dim(kerA N rangeB). 

If F = C, then rank(AB) = rank(A) — dim(kerB* M rangeA*). 

Multiplication ofa matrix from the left or right by an invertible matrix leaves the rank unchanged. 
null(AB) = null(B) + dim(kerA NM rangeB). 

rank(AB) < min{rank( A), rank(B)}. 

rank(AB) > rank(A) + rank(B) — n. 


. (Sylvester’s law of nullity) Let A, B € C”*". Then 


max{null(A), null(B)} < null(AB) 
< null(A) + null(B). 


The above fact is valid only for square matrices. 


. (Frobenius inequality) Let A €¢ F™”*", B ¢ F"**, and C € F**?, Then 
q y 


rank(AB) + rank(BC) < rank(B) + rank(ABC). 


. Let Ac C”*". Then 


rank(A* A) = rank(A) = rank(AA*). 
In fact, 
range(A*A) = rangeA* and _ rangeA = range(AA*), 
as well as 


ker(A* A) =kerA and _ ker(AA*) = kerA*. 


. Let Ae F™*" and B € F**?. The rank of their direct sum is 


A 


rank(A ® B) = rank 0 8B 


= rank(A) + rank(B). 


. LetA = [ay] ¢ F"" and Be F*x?, The rank of the Kronecker product A@ B = [ajB] € Fmkxnp 


1s 


rank(A ® B) = rank(A)rank(B). 
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14, Let A = [ay] € F™*" and B = [by] € F”*". The rank of the Hadamard product Ao B = 
[abi] € F'*” satisfies 


rank(A o B) < rank(A)rank(B). 


Examples: 


1. Consider the matrices 


1 -1 1 2 3 4 Le 2d od 
A=|]2 -1 0], B=/0 0 -1], and C=]1 2 -1 1 
So, =2: 1 2-1 2 2. “AL 2: 2 


We have that 


rank(A)=2, rank(B)=3, rank(C)=1, rank(A+ B) =3, 
rank(AB) = 2, rank(BC)=1, rank(ABC) = 1. 


As a consequence of Fact 5, we have 


null(A) =3—2=1, null(B)=3-—3=0, null(C)=4-1=3, 
null(A+ B)=3—3=0, null(AB)=3-—-2=1, 
null(BC) =3—1=2, null(ABC) =4-1=3. 

Fact 6 states that 


—1=2-—3=rank(A) — rank(B) < rank(A + B) = 0 < rank(A) + rank(B) = 5. 
Since rangeA M rangeB = rangeA, Fact 7 states that 


rank([A|B]) = rank(A) + rank(B) — dim(rangeA M rangeB) = 2+ 3—2 = 3, 
null([A|B]) = null(A) + null(B) + dim(rangeA M rangeB) = 1+0+4+2=3. 


Since kerA M rangeB = kerA, Fact 8 states that 
2 = rank(AB) = rank(B) — dim(kerA/N rangeB) = 3 — 1 =2. 


2 = rank(AB) < min{rank(A), rank(B)} = 2. 
2 = rank(AB) > rank(A) + rank(B) -n=24+3-—-3=2. 


Fact 9 states that 


1 = max{null(A), null(B)} < null(AB) = 1 
< Null(A) + null(B) = 1. 


Fact 9 can fail for nonsquare matrices. For example, if 
D=[1 1), 
then 


1 = max{null(D), null(D")} £ null(DD‘) = 0. 
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* Fact 10 states that 


3 = rank(AB) + rank(BC) < rank(B) + rank(ABC) = 4. 


Useful Identities for the Inverse 


This section presents facts and formulas related to inversion of matrices. 


Facts: 
Ie 


[Oue81, (1.9)], [HJ85, p. 18] Recall that A/ A[a] denotes the Schur complement of the principal 
submatrix A[@] in A. (See Section 4.2 and Section 10.3.) If A € F"*" is partitioned in blocks as 


Au Ai 


A= 
Ar Az 


> 


where Aj; and A») are square matrices, then, provided that A, Aj, and A» are invertible, we have 
that the Schur complements A/A,; and A/ Az, are invertible and 


(A/Ax2) — Aj! Ayp(A/An)7! 


Al= z a 
—(A/An) An At (A/A\\)7! 


More generally, given an invertible A € F”*”" anda C {1,2,...,n} such that A[a] and A(q) are 
invertible, A~! is obtained from A by replacing 

* Ala] by (A/A(a))“, 

* Ala,a°] by —A[a]~! Ala, a°](A/A[a]), 

* Ala‘,a] by —(A/A[a])~' Ala’, a] A[o]~!, and 

* A(a) by (A/A[@])7!. 


. [HJ85, pp. 18-19] Let A € F"*", X € F™', R © F’*", and Y € F’*". Let B= A+ XRY. 


Suppose that A, B, and R are invertible. Then 


BY =(A+ XRY)'=A!—ATX(R'NFYATX)TYAT. 


. (Sherman-Morrison) Let A € F"*", x,y € F". Let B = A+<xy?. Suppose that A and B are 


invertible. Then, if y’ A7!x 4 —-1, 


1 
Bo -(A Ty-1 = At Avi Taml. 
(A+ xy") ia 
In particular, if yl x # —1, then 
(I+xy?)t=1- ok gt 
l+yTx * ° 


. Let A € F”"*". Then the adjugate of A (see Section 4.2) satisfies 


(adjA)A = A(adjA) = (det A)I. 


If A is invertible, then 
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5. Let A € F"*" be invertible and let its characteristic polynomial be p(x) = x" + dy_1x"! + 
An—2x" 7? + +++ + a,x + do. Then, 


-1 as n+1 n n—-1 
Ae = 4A (A"™ +a, A" + a, A" ~ +++++4,_1A). 


6. [Mey00, Sec. 7.10, p. 618] Let A € C”*”. The following statements are equivalent. 
* The Neumann series, [+ A+ A?+..., converges. 
CO 
* (I — A)! exists and (I — A)7! = oy A‘. 
k=0 
* p(A) <1. 
* lim A‘ =0. 
k>0o 


Examples: 


1. Consider the partitioned matrix 


1 3)-1 
A= - i elt Ge elle ay 
21 A22 i =e of 


Since 


-1 
(ayaa = [ 4 Sls j and (A/Au)7! = (-1)7! =-1, 


by Fact 1, we have 


(A/Ax) —Ajq! Ap(A/An)7! 


Al= z 
—(A/Ay)7! An Aq (A/Ay,)7! 


2. To illustrate Fact 3, consider the invertible matrix 


1 # =I 
A= 1 0 1 
—2i 1 -2 


and the vectors x = y = [111]?. We have that 


0.51 1+0.5i 0.5 
Al=]-1-i -1+i i 
0.5% 0.5% 0.5 


Adding xy? to A amounts to adding 1 to each entry of A; since 


l+y?Aly =i + 0, 


Matrix Equalities and Inequalities 14-17 


the resulting matrix is invertible and its inverse is given by 


1 
(A+xy7)7! = Av = A ey™ AT 
1 


2.5 0.5 — 0.51 1-i 
= |-2+2i 1 2 
—1.5-i1 0.5 + 0.52 i 
3. Consider the matrix 
—l 1 -l 
A= 1 -l 3 
1 -l 2 


Since A? = 0, A is a nilpotent matrix and, thus, all its eigenvalues equal 0. That is, o(A) = 0 < 1. 
As a consequence of Fact 6, J — A is invertible and 


1 o1 
(I-A) t=I+A4+A7=]2 -1 5 
1 -1 3 
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There is a vast amount of material in matrix (operator) perturbation theory. Related books that are worth 
mentioning are [SS90], [Par98], [Bha96], [Bau85], and [Kat70]. In this chapter, we attempt to include the 
most fundamental results up to date, except those for linear systems and least squares problems for which 
the reader is referred to Section 38.1 and Section 39.6. 

Throughout this chapter, || - |p. denotes a general unitarily invariant norm. Two commonly used ones 
are the spectral norm || - ||, and the Frobenius norm || - ||;. 


15.1 Eigenvalue Problems 


The reader is referred to Sections 4.3, 14.1, and 14.2 for more information on eigenvalues and their 
locations. 


Definitions: 


Let A € C”"*". A scalar—vector pair (A,x) € C x C” is an eigenpair of A if x #4 0 and Ax = Ax. A 
vector—scalar—vector triplet (y,4,x) € C” x C x C” is an eigentriplet if x # 0, y # 0, and Ax = Ax, 
yA = dy*. The quantity 


nadie Ixllallyll 


ly*x| 


is the individual condition number for 1, where (y,4,x) € C” x C x C” is an eigentriplet. 
Let o(A) = {Aj,A2,... An}, the multiset of A’s eigenvalues, and set 


A= diag(A, A2; tee An)» Ay = diag(Az(1),Ar(2)s tee sAr(n))s 
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where Tt is a permutation of {1,2,... ,m}. For real A, ie., all 1;’s are real, 
At =diag(al,ad,...,at). 


A? is in fact a A, for which the permutation t makes A,(;) = at for all j. 


Given two square matrices A; and A), the separation sep(A,, Az) between A, and Aj is defined as [SS90, 
p. 231] 


sep(A;, Az) = inf || XA, — A)X|l2. 
IXI2=1 
A is perturbed to A= A+AA. The same notation is adopted for A, except all symbols with tildes. 


Let X, Y € C’** with rank(X) = rank(Y) = k. The canonical angles between their column spaces are 
6; = arc coso;, where {oi}, are the singular values of (Y*Y)7!/*y* X(X*X)—!/?, Define the canonical 
angle matrix between X and Y as 


O(X, Y) = diag(6,,0:,... Or). 


For k = 1, i.e., x, y € C” (both nonzero), we use Z(x, y), instead, to denote the canonical angle between 
the two vectors. 


Facts: 
1. [SS90, p. 168] (Elsner) max min |A; — Aj] < (IJAll2 + All) "A All”. 
ij 
2. [SS90, p. 170] (Elsner) There exists a permutation t of {1,2,... ,} such that 
x n 4 .\l-1/n I/n 
|A — Arllz <2 5 (IAll2 + Alla) |AAll,"”. 
3. [SS90, p. 183] Let (y, jz, x) be an eigentriplet of A. AA changes jz to uw + Ap with 
A)x > 
An = aa, O(|AAll3), 


and |Aj.| < cond()|| AAll2 + O(||AAll3). 
4. [SS90, p. 205] If Aand A + AA are Hermitian, then 


At — At < AAllu- 


5. [Bha96, p. 165] ( Hoffman—Wielandt) If Aand A+ A Aare normal, then there exists a permutation 
t of {1,2,...,”} such that || A — A;|l; < |JAAll:. 


6. [Sun96] If A is normal, then there exists a permutation t of {1,2,... ,m} such that ||A — A, lly < 
Jnl|A Alle. 

7. [SS90, p. 192] (Bauer—Fike) If A is diagonalizable and A = XAX7! is its eigendecomposition, 
then 


max min |Aj — Aj] < | X7'(AA)X |p < Kp(X)|AAllp. 
f J 


8. [BKL97] Suppose both A and A are diagonalizable and have eigendecompositions A = XAX7! 
and A= XAX7!, 


(a) There exists a permutation Tt of {1,2,... ,} such that 


A — Agile < Ve2(X)e2(X) JAAlle- 


(b) At — Aten < \/e2(X)K2(X)||A Aller for real A and A. 
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9. [KPJ82] Let residuals r = Ax — f#X and s* = y*A — jy*, where ||X||2 = |lyll2 = 1, and let 
€ = max {|lr|l2, ||s||2}. The smallest error matrix A A in the 2-norm, for which (jy, 2, x) is an exact 
eigentriplet of A= A+ AA, satisfies || AAl|> = €, and |Z — pu| < cond(jt) e + O(e?) for some 
pe €a(A). 

10. [KPJ82], [DK70],[Par98, pp. 73, 244] Suppose A is Hermitian, and let residual r = Ax — x and 
|xll2 = 1. 
(a) The smallest Hermitian error matrix A A (in the 2-norm), for which (jz, x) is an exact eigenpair 
of A= A+ AA, satisfies || AAll2 = |Jrll2. 


(b) | — “| < |[rl|2 for some eigenvalue pu of A. 
(c) Let yz be the closest eigenvalue in 0 (A) to pz and x be its associated eigenvector with ||x||2 = 1, 


andletn = min | —Al|.Ifn > 0, then 
utheo(A) 


2 
a r . 5 r 
i) ee, inex < ME. 
n 


11. Let A be Hermitian, X € C"** have full column rank, and M € C*** be Hermitian having 
eigenvalues j4) < 2 <--- < py. Set 
R= AX — XM. 


There exist k eigenvalues Aj, < Ai, <--++ < Aj, of A such that the following inequalities hold. Note 
that subset {A,, Yaa may be different at different occurrences. 


(a) [Par98, pp. 253-260], [SS90, Remark 4.16, p. 207] (Kahan—Cao-Xie-Li) 


IRll2 
Omin(X) , 


max |; — Aj,| < 
reas il < 


k 
Rll: 
(uj —Ai,)? S ; 
» : : Omin(X) 


(b) [SS90, pp. 254-257], [Sun91] If X*X = I and M = X* AX, and if all but k of A’s eigenvalues 
differ from every one of M’s by at least 7 > 0 and ¢,; = || R||;/n < 1, then 


IRI? 


k 
(te — A4,)? < ——_. 
X a tape 


(c) [SS90, pp. 254-257], [Sun91] If X*X = I and M = X* AX, and there is a number 7 > 0 such 
that either all but k of A’s eigenvalues lie outside the open interval (44; — 9, “x + 7) or all but 
k of A’s eigenvalues lie inside the closed interval [we + 1, 4e+1 — n] forsome 1 < £<k—1, 
and ¢ = ||R|l2/n < 1, then 


12. [DK70] Let A be Hermitian and have decomposition 


Al 


A[X, X2] = Ay 


* 
1 
* 
2 
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where [X, X] is unitary and X; € ck Let Q € C”** have orthonormal columns and for ak x k 
Hermitian matrix M set 


R= AQ- QM. 


Let 7 = min | — v| over all w € o(M) and v € o(A2). If n > 0, then || sin @(X1, Q)||; < 
13. [LLO5] Let 


Rll 
U 


be Hermitian, and set 7 = min | — v| over all ~ € o(M) and v € o(H). Then 
2IE Il 
nt a/n? + ll EIl3 


14. [SS90, p. 230] Let [X, Y2] be unitary and X; € cr, and let 


max |A! —Al| < 
l<j<n / J 


A; G 


A[X, Y2] = 
[Xi Yo] E A 


* 
1 
* 
Y 


Assume that o(A;)(\o(A2) = Y, and set n = sep(Aj, A). If ||G|l2||E|l2 < 7/4, then there is a 
unique W € C&"—-hxk, satisfying ||W||2 < 2||E||2/n, such that [X, Y>] is unitary and 


where 
X,=(X%1+ YW) + ww), 
Y) = (Y. — X,W*)(I + Ww*)-?, 
A, = (I + W*W)!/2(A, + GW) + W*W)7!?, 
Ay =(1 + WW*)712(4, — WG)(IT + WW*)!??, 


= 2IE 
Thus, || tan @(X), Xi)|l2 < NE fa 
n 


Examples: 
1. Bounds on ||A — A;||u: are, in fact, bounds on ij — Ax(j) in disguise, only more convenient and 
concise. For example, for || - lu: = || - 2 (spectral norm), ||A — A;z||2 = max; |Aj — Axcj)|, and for 
: ~ 1/2 
ll - llur = Il - lly (Frobenius norm), || A — Az|lp = i= |Aj = Ati l? 


2. Let A, Ae C"™" as follows, where ¢ > 0. 


wool wool 
ies lL = bu 
1 eee 
bL é lb 
It can be seen that o(A) = {u,...,p} (repeated n times) and the characteristic polynomial 


det(tl — A) = (t — w)" — «, which gives o(A) = {uw + €!/"e7i7/",0 < j < n— 1}. Thus, 
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6. 


A —pl|=el/" = | AAlly/”. This shows that the fractional power || AAs!” in Facts 1 and 2 cannot 
be removed in general. 


Consider 
1 2 3 0 0 0 
A=|]0 4 5 isperturbedby AA=] 0 0 0 
0 0 4.001 0.001 0 0 


A’s eigenvalues are easily read off, and 


dy = 1, x, = [1,0,0]’, y; = [0.8285, —0.5523, 0.0920]", 
do = 4, X) = [0.5547, 0.8321,0]", yo = [0, 0.0002, —1.0000]’, 


3 = 4.001, x; = [0.5547, 0.8321, 0.0002]7, y; = [0,0, 1]. 


On the other hand, A’s eigenvalues computed by MATLAB’s eig are re = 1.0001, o = 3.9427, 
A3 = 4.0582. The following table gives | ; — 4.;| with upper bounds up to the Ist order by Fact 3. 


cond(aj) cond(Aj)IAAll2 1A; — A; 


j 

1 1.2070 0.0012 0.0001 
2 6.0-10° 6.0 0.057 
3 6.0- 10° 6.0 0.057 


We see that cond(A ;)|| AAll2 gives a fairly good error bound for j = 1, but dramatically worse for 
j = 2, 3. There are two reasons for this: One is in the choice of AA and the other is that A A’s 
order of magnitude is too big for the first order bound cond(A ;)|| A All2 to be effective for j = 2, 3. 
Note that AA has the same order of magnitude as the difference between i and A3 and that is too 
big usually. For better understanding of this first order error bound, the reader may play with this 


example with AA = ¢ “1 for various tiny parameters ¢. 
lly; lla IX Ile 
. Let & = diag(c),c2,... ,cx) and P = diag(s),52,... ,5,), where cj,s; > 0 and ci aS = | forall 


j. The canonical angles between 


I x 
X=Q/0/V*, Y=Q{Ir} U* 
0 0 
are 6; = arccoscj, j = 1,2,...,k, where Q, U, V are unitary. On the other hand, every pair 


of X, Y € C"** with 2k < nand X*X = Y*Y = k, having canonical angles arccos cc ;, can be 
represented this way [SS90, p. 40]. 


. Fact 13 is most useful when || E ||2 is tiny and the computation of A’s eigenvalues is then decoupled 


into two smaller ones. In eigenvalue computations, we often seek unitary |X, X2] such that 


ou M 0 
A[X, X2] = i al 


* x 


M_ E* 
E HA 


> 


A(X, X2)] = | 


1 1 
xy xy 
and ||E || is tiny. Since a unitarily similarity transformation does not alter eigenvalues, Fact 13 still 
applies. 

[LLO5] Consider the 2 x 2 Hermitian matrix 
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where aw > 6 and e > 0. It has two eigenvalues 


i aE Bs By? ae 
+ 2 > 


and 


Aya 267 
raed oar B) + V/(a— BY + 467” 


The inequalities in Fact 13 become equalities for the example. 


15.2 Singular Value Problems 


The reader is referred to Section 5.6, Chapters 17 and 45 for more information on singular value decom- 
positions. 


Definitions: 


B € C”™*" has a (first standard form) SVD B = UX V*, where U € C”™*” and V € C”™” are unitary, 
and & = diag(o),02,...) € R”*” is leading diagonal (0; starts in the top-left corner) with all oj > 0. 

Let SV(B) = {01,02,... ;Omin{mn}}, the set of B’s singular values, and a; > 02 > --- > 0, and let 
SVext(B) = SV(B) unless m > n for which SVext(B) = sv(B) U{0,... , 0} (additional m — n zeros). 

A vector—scalar—vector triplet (u,o,v) € C” x R x C" isa singular-triplet if u 4 0, v 4 0,0 > 0, and 
Byv=ou, B*u=ov. 

B is perturbed to B = B + AB. The same notation is adopted for B, except all symbols with tildes. 


Facts: 
1. [SS90, p. 204] (Mirsky) || XZ — Dllur < JABlu- 
2. Let residuals r = BV — ands = B*u — Lv, and ||v||. = |/ul|, = 1. 
(a) [Sun98] The smallest error matrix AB (in the 2-norm), for which (u, (2, V) is an exact singular- 
triplet of B = B + AB, satisfies || AB ||, = ¢, where € = max {|r|l2, ||s|l2}. 
(b) | — “| < € for some singular value yu of B. 
(c) Let yp be the closest singular value in SVex¢(.B) to 4 and (u, o, v) be the associated singular-triplet 
with ||ul|2 = ||vl]2 = 1, and let 7 = min|fZ — o| over allo € SVex(B) ando ¥ w.Ifn > 0, 
then |i — p| < e7/n, and [$S90, p. 260] 
= = Vv Wells + lsil3 
‘sin? Z(u,u) + sin? Z(v,v) < ea sr 
UT] 
3. [LLO5] Let 


B, F 
B — 
E B, 
where B, € C**, and set 7 = min|jz — v| over all w € SV(By) and v € SVex(Bo), and ¢ = 
max{|| E |l2, || F |l2}- Then 


ge on 10 
ec™™", B= : 
0 Bs 


267 


nt VP + 4e2 


J 
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4. [SS90, p. 260] (Wedin) Let B, Bec (m > n) have decompositions 


‘ B, 0 


BIMVl=|q 


> 


2 
where [U; U2], [Vi V2], [U; U5], and [V; Vo] are unitary, and U), U; eck V,, Vi € C"*k Set 
R=BV,—U,B), S = BYU, — ViBi. 


If sv(By) (1) SVext( Bz) = Y, then 


— — Riz S12 
Visine, DoF + isinovs, WR s SERRE, 


where 7 = min |j — v| over all jf € sv(By) and v € SVext( Bz). 


Examples: 
1. Let 


. —3 ~ cf 
fa ef malt JEG 
2 


Then o; = 2.000012, 02 = 0.999988, and a; = 2, 0 = 1. Fact 1 gives 


2 

max |oj —G|< 4-10, | oj —G|? <5-10°. 
1<j<2 I 
Ve 


2. Let B beas in the previous example, and let V = e, U = e2, ff = 2. Thenr = Bv—jiu = 3-107 e, 
ands = B*u — fiv = 4- 10~%e). Fact 2 applies. Note that, without calculating sv(B), one may 
bound 77 needed for Fact 2(c) from below as follows. Since B has two singular values that are near 1 
and jt = 2, respectively, with errors no bigger than 4- 1073, then n > 2—(1+4-1077) = 1—4-107°. 


3. Let B and B beas in Example 1. Fact 3 gives max |oj —oj| < 1.6- 10>, a much better bound 
<j< 
than by Fact 1. 


4, Let B and B be as in Example 1. Note B’s SVD there. Apply Fact 4 with k = 1 to give a similar 


bound as by Fact 2(c). = 
5. Since unitary transformations do not change singular values, Fact 3 applies to B, B € C”*" having 
decompositions 
U; B, F Uy] x B, 0 
B[V, Vy] = 5. B[V, Val = > 
E MiM=le al lye| FM MALO a, 


where [U, U2] and [V, V2] are unitary and U, € cm yy e Cm, 


15.3 Polar Decomposition 


The reader is referred to Chapter 17.1 for definition and for more information on polar decompositions. 


Definitions: 


B € F”*" is perturbed to B = B + AB, and their polar decompositions are 
B=QH, B= QH=(Q+AQ)(H+ AH), 


where F = R or C. AB is restricted to F for B € F. 
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Denote the singular values of B and Bas 0, > 02 >--- ando| > 2 > ---, respectively. 
The condition numbers for the polar factors in the Frobenius norm are defined as 


AX 
condg(X) = lim WAX Te 


, forX=HorQ. 
5>0 aBip<s 6 


B is multiplicatively perturbed to BifB= DB Dy for some D, € F”*” and D, € F"*". 

B is said to be graded if it can be scaled as B = GS such that G is “well-behaved” (i.e., k2(G) is of 
modest magnitude), where S is a scaling matrix, often diagonal but not required so for the facts below. 
Interesting cases are when k2(G) & k2(B). 


Facts: 
1. [CGO00] The condition numbers condg(Q) and condp(H) are tabulated as follows, where x2(B) = 


O71 /On- 


R C 
FactorQ m=n_  2/(On-1+0n) 1/On 


m>n 1/on 1/on 


2(1 + K2(B)*) 


FactorH m>n 
1+ k2(B) 


2. [Kit86] || AH||; < /2|| AB||:. 
3. [Li95] Ifm = n and rank(B) = n, then 


[AQllu | AB|lu- 


n On 


4. [Li95], [LSO2] Ifrank(B) = n, then 


| AQ|lu S ( = ) AB lu, 


On + on max{o,,, On} 


AQ: < 


| AB Ilr. 


n n 


5. [Mat93] If B € R"*”, rank(B) = n, and || AB||2 < o,, then 


2||AB AB 
[AQIu = SAEs. (1 4 at) 
| AB Il, On + Ont 
where || - |, is the Ky Fan 2-norm, i.e., the sum of the first two largest singular values. (See 


Chapter 17.3.) 
6. [LS02] If B € R”*”, rank(B) = n, and ||AB||2 < o, +4,, then 


4 
A < AB lp. 
|AQIle < Aa ee ABs 


7. {Li97] Let B and B= D* BD, having full column rank. Then 


|AQIl: < vil Dele elle Ae vu = Dele + lLPa =z 


8. [Li97], [Li05] Let B = GS and B = GS and assume that G and B have full column rank. If 
| AG]2I|G" 2 < 1, then 
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|AQlls Sv 1G" al AGIs, 
AHS“ Mle < (y GaGa +1) IAG 


where y = V1 += Gt] AGIl2) 


Examples: 


1. Take both B and B to have orthonormal columns to see that some of the inequalities above on AQ 
are attainable. 
2. Let 


pe | 201 502}_ 1 fl 1 Hr. 22 
~ /2 |-1.99 -498] — /2]1 -1 2 5-10? 


~ | 1.4213 3.5497. 10? 
~ 11.4071 —3.5214- 107 


and 


obtained by rounding each entry of B to have five significant decimal digits. B = QH can be read 
off above and B = QH can be computed by Q = UV* and H = V2 V*, where B’s SVD is 
UXV*. One has 


sv(B) = {5.00 - 102, 2.00 - 1073}, sv(B) = {5.00 - 10, 2.04 - 1077} 


and 


WABll2 ABI = AQII2 TAQ TAH Il2 TA Alle 


3-1073 93-1073 = 2-107 = 33-10 3S 22-1073 — 2- 10-9 


Fact 2 gives || AH||; < 3- 107? and Fact 6 gives || AQ||; < 107°. 
3. [Li97] and [Li05] have examples on the use of inequalities in Facts 7 and 8. 


15.4 Generalized Eigenvalue Problems 


The reader is referred to Section 43.1 for more information on generalized eigenvalue problems. 


Definitions: 


Let A, B € C”*". A matrix pencil is a family of matrices A — AB, parameterized by a (complex) number 
i. The associated generalized eigenvalue problem is to find the nontrivial solutions of the equations 


Ax=)Bx and/or y*A=Ay'B, 


wherex € C’,ye C”, anda €C. 

A — 2B is regular if m = n and det(A — AB) # 0 for some A € C. 

In what follows, all pencils in question are assumed regular. 

Aneigenvalue d is conveniently represented by a nonzero number pair, so-called a generalized eigenvalue 
(a, B), interpreted as A = a/f. B = 0 corresponds to eigenvalue infinity. 
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A generalized eigenpair of A — 1B refers to ((a, B),x) such that BAx = a Bx, where x 4 0 and 
|a|? + ||? > 0. A generalized eigentriplet of A — 1B refers to (y, (a, B),x) such that B Ax = w Bx and 
By* A = ay* B, where x 4 0, y ¥ 0, and |a|? + |B|? > 0. The quantity 


IxIl2llyll2 
J/ly* Axl? + ly* Bx|? 


cond((q, B)) 


is the individual condition number for the generalized eigenvalue (a, 8), where (y, (a, 6), x) isa generalized 
eigentriplet of A— AB. 

A — XB is perturbed to A—AB =(A+ AA) —X(B + AB). 

Let o(A, B) = {(a, Bi), (2, Bo)... - 5 (ns Bn)} be the set of the generalized eigenvalues of A— 1B, and 
set Z =[A, B] € C2"*", 

A—AB is diagonalizable if it is equivalent to a diagonal pencil, i.e., there are nonsingular X, Y € C”*" 
such that Y*AX = A, Y*BX = Q, where A = diag(a1,02,... ,@,) and Q = diag(B), B2,... 5 Bn). 

A — iB isa definite pencil if both A and B are Hermitian and 


y(A,B)= min |x*Ax+ix*Bx| > 0. 


xeC",||x|]2=1 


The same notation is adopted for A — 0B, except all symbols with tildes. 
The chordal distance between two nonzero pairs (a, 8) and (q@, B) is 


|Bo — &B| 
Vial? + IB? lel? + IB 


x ((a, B), (@, B)) = 


Facts: 


1. [SS90, p. 293] Let (y, (a, 8), x) bea generalized eigentriplet of A—1B.[AA, AB] changes (a, 8) = 
(y* Ax, y* Bx) to 


(a, B) = (a, B) + (y*(AA)x, y*(AB)x) + O(e7), 


where ¢ = ||[AA, AB]|l2, and x ((a, B), (@, B)) < cond((a, B)) ¢ + O(¢?). 
2. [SS90, p. 301], [Li88] If A — 1B is diagonalizable, then 


max min x ((a, Bi), (Bj) < ka(X)|| sin O(Z*, Z|] 


3. [Li94, Lemma 3.3] (Sun) 


IZ — Zllu 
max{Omin(Z), Omin(Z)} 


|| sin @(Z*, Z*)llu < 


where Omin(Z) is Z’s smallest singular value. 

4. The quantity y(A, B) is the minimum distance of the numerical range W(A + 7B) to the origin 
for definite pencil A — AB. 

5. [SS90, p. 316] Suppose A — 1B isa definite pencil. If A and B are Hermitian and ||[AA, AB]]l2 < 
y (A, B), then A—AB isalso a definite pencil and there exists a permutation Tt of {1,2,... ,m} such 
that 


Pere ((a; Bi) ( es B. »)) < |[AA, AB] |l2 
lejen” PRI =(j)> i) = y(A, B) 


6. [SS90, p. 318] Definite pencil A — 1B is always diagonalizable: X* AX = A and X*BX = Q, and 
with real spectra. Facts 7 and 10 apply. 
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7. [Li03] Suppose A — AB and A— AB are diagonalizable with real spectra, i.e., 


10. 


Y*AX=A,Y*BX=Q and Y*AX=A, Y*BX=Q, 
and all (aj, 8;) and all (@;, B j) are real. Then the follow statements hold, where 
a= diag(x (a1, B1); (G21) Bray))s tee X Cen Bn); (Qz(n)> Brin)))) 


for some permutation Tt of {1,2,... ,n} (possibly depending on the norm being used). In all cases, 
the constant factor 1/2 can be replaced by 1 for the 2-norm and the Frobenius norm. 


a Ai ie 
(a) ||Sllu < 5 VV K(X )k2(X)|| sin ©(Z*, Z*) lu. 


(b) Ifall Ja; |? + |B; |? = la; |? + |B; |? = 1 in their eigendecompositions, then 
j j j j 8g P 
WSllu < 5 = /XUlollY*llall Xa Y*laAA, AB] |lu- 


. Let residuals r = 6 AX — @BX and s* = By* A — ay*B, where ||X||, = ||yll2 = 1. The smallest 


error matrix [AA, AB] in the 2-norm, for which (y, (@, 8),x) is an exact generalized eigentriplet 
of A — 2B, satisfies ||[AA, AB]|l, = ¢, where ¢ = max {([rll2, Ilsll2}, and x ((a, B), (a, B)) < 
cond((@, B)) ¢ + O(e*) for some (a, B) € o(A, B). 


. [BDD00, p. 128] Suppose A and B are Hermitian and B is positive definite, and let residual 


t= Ax — Bx and ||x||, = 1. 
(a) For some eigenvalue jz of A — AB, 


Gi — | < Willen 


—l 
— Bh SS SB" Malltlle 
IxIlB 


where ||z|| 7 = /z* Mz. 


(b) Let yw be the closest eigenvalue to 4 among all eigenvalues of A — AB and x its associated 
eigenvector with ||x||, = 1, and let 7 = min|j — v| over all other eigenvalues v 4 pw of 
A—AB.Ifn > 0, then 


- 1 (Url? Il? 
i — pl < ( a, ) <a 
a \ [lle 


= r 
sin ZG) = Bp/2eaB) SE. 
Ul 


[Li94] Suppose A — AB and A—AB are diagonalizable and have eigendecompositions 


x * 


Ay 
Ad 


Qy 


A[X,, Xo] = Q® 


> B[X, X)] = 5) 


1 1 
x x 
Y; Y; 


x7 =[W, Wa)", 


and the same for A— AB except all symbols with tildes, where X1, Y;, W, € C”™**, Ay, Q) € CH**, 
Suppose |a;|? + ie ? = jar + 1B; ) =1forl < j < ninthe eigendecompositions, and set 
n = min x ((a, B), (a, B) ) taken over all (@, 8) € o(Aj,Q1) and (@, B) € o (Ag, Q3). Ifn > 0, 
then 


X42 Wl. 
1) 


X| 


Y(Z=Z) se 


sin (x1, X1)| = 
F 


F 


15-12 Handbook of Linear Algebra 


15.5 Generalized Singular Value Problems 


Definitions: 


Let A € C”*%" and B € C**". A matrix pair {A, B} is an (m,£,n)-Grassmann matrix pair if 
A 
rank B =n. 
In what follows, all matrix pairs are (m, £,n)-Grassmann matrix pairs. 
A pair (a, 8) is a generalized singular value of { A, B} if 


det(B?.A* A — a7 B*B) = 0, (a, B) £ (0,0), a, B > 0, 


ie., (a, B) = (./, /V) for some generalized eigenvalue (1, v) of matrix pencil A* A — AB*B. 
Generalized Singular Value Decomposition (GSVD) of {A, B}: 


U*AX=%,4, V*BX =p, 


where U € C”™™™, V € C™ are unitary, X € C”*” is nonsingular, ©, = diag(a,a2,---) is leading 
diagonal (a; starts in the top left corner), and Xg = diag(--- , Bn—1, Bn) is trailing diagonal (6; ends in 
the bottom-right corner), aj, Bj = 0 and ae + BF = 1for1 < j <n. (Set some a; = 0 and/or some 
B; = 0, if necessary.) oe 

{ A, B} is perturbed to {A, B} = {A+ AA, B + AB}. 

Let svV(A, B) = {(a1, Bi), (02, B2),... 5 (Ons Bn)} be the set of the generalized singular values of {A, B}, 


and set Z = : € Cm Oxn 


The same notation is adopted for { A, B}, except all symbols with tildes. 


Facts: 


1. If {A, B} is an (m, £,n)-Grassmann matrix pair, then A* A — A B*B is a definite matrix pencil. 
2. [Van76] The GSVD of an (m, €,n)-Grassmann matrix pair { A, B} exists. 
3. [Li93] There exist permutations t and of {1,2,... ,m} such that 


max x ((a;; Bj); (&r(i)» Bray) < Il sin O(Z, Z) Ilo, 


l<j<n 


4. [Li94, Lemma 3.3] (Sun) 


IZ — Zhu 
max{Omin(Z), Omin(Z)} ; 


|| sin @(Z, Z)llu < 

where Omin(Z) is Z’s smallest singular value. 

5. [Pai84] If a? + B? = a? + B? = 1 fori = 1,2,...,n, then there exists a permutation w of 
{1,2,...,n} such that 


» l(e; = Hey(j))? + (Bj — Batj)?| S an i — Zo Qllrs 


where Zp = Z(Z*Z)~'/? and Zp) = Z(Z*Z)~"/?. 
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6. [Li93], [Sun83] Perturbation bounds on generalized singular subspaces (those spanned by one or 
a few columns of U, V, and X in GSVD) are also available, but it is quite complicated. 


15.6 Relative Perturbation Theory for Eigenvalue Problems 


Definitions: 


Let scalar w be an approximation to a, and 1 < p < oo. Define relative distances between a and @ as 
follows. For |a|* + |a|* 4 0, 


d(a,a@) = me 1 = ae a} classical measure) 
a |o| 
la—a| F 
(a, @) = ———__—., [Li98]) 
ss Vial? + lal? 
. |a-a| 
(a, @) = ——, [BD90], [DV92]) 
Vv |aa| 
s(a,a@)=|In(@/a)|, for@a>0, ([LM99a], [Li99b]) 


and d(0,0) = @p(0,0) = ¢(0,0) = ¢(0,0) Os pb: 

A € C"*" is multiplicatively perturbed to A if A = D* AD, for some D,, Dz € C"*". 

Denote o(A) = {Aj,A2,... » An} and o(A) = pve as dea 

A € C"*" is said to be graded if it can be scaled as A = S*HS such that H is “well-behaved” 
(i.e., K2(.H) is of modest magnitude), where S is a scaling matrix, often diagonal but not required so for 
the facts below. Interesting cases are when k2(H) < k2(A). 


Facts: 


1. [Bar00] @p(-, -) isa metric on C forl1 < p< om. 
2. Let A, A = D* AD € C”*" be Hermitian, where D is nonsingular. 


(a) [HJ85, p. 224] (Ostrowski) There exists t;, satisfying 
Amin( D* D) < tj as Amax(D* D), 
such that al = jal for j = 1,2,... ,n and, thus, 


max d(A},A}) < |I — D*Dllo. 


l<j<n 


(b) [LM99], [Li98] 


Iidiag (c(A1,21),.-..6(1,42)) ler S IHIn(D* D)llun 


Iidiag (¢A1,2]),... ¢01,21)) lla S Dt — Dla. 


3. [Li98], [LM99] Let A = S*HS be a positive semidefinite Hermitian matrix, perturbed to 
A= S*(H + AH)S. Suppose H is positive definite and || H~!/?(AH)H~!/?||, < 1, and set 
M= H'?ss*H"?,/ M= DMD, 
where D = [I + H7'/?(AH)H~"?]'? = D*. Then o(A) = o(M) and o(A) = o(M), and the 
inequalities in Fact 2 above hold with D here. Note that 
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We < |H-?(AH) HO? Iu 
oo /1— H(A) A? 5 
Ho" Il2 
~ J1— TA hIAAT 


||D— D™ 


|A Alu. 


4. [BD90], [VS93] Suppose A and Aare Hermitian, and let | A| = (A?)!”” be the positive semidefinite 
square root of A’. If there exists 0 < 5 < 1 such that 


|x*(A A)x| < 6x*|A|x forallx € C’, 


then either 41 = at =O0orl—d< ayy <1+6. 
5. [Li99a] Let Hermitian A, A = D*AD have decompositions 


x yx od 
1 _ Ar Xil~s = — {Ai 
xt A[X, X2] = Al? ce A[X, X2] = i ) 
where [X, X>] and [xa eH are unitary and X), x eC™* Ifm = min — Q2(,f) > 0, 


wEo(A;), “Eo(A2) 
then 


VIE = PDX AIG = DY A2 
2 


Il sin O(X1, Xe < 


6. [Li99a] Let A= S* HS bea positive semidefinite Hermitian matrix, perturbed to A=S *(H + AH)S, 
having decompositions, in notation, the same as in Fact 5. Let D = [I + H (ARH | ue 


Assume H is positive definite and ||H~/?(AH)H~'?||, < 1. Ifne = min — C(u,p) > 0, 
peo (Aj), WEo(A2) 


then 
. = ||D — D7" 
|| sin O(X,, X1)||y < —————.. 
Nc 
Examples: 


1. [DK90], [EI95] Let A be a real symmetric tridiagonal matrix with zero diagonal and off-diagonal 
entries b), b2,... ,b,—1. Suppose A is identical to A except for its off-diagonal entries which change 
to Bi by, Bobo,... ; By—1bn—1, where all 6; are real and supposedly close to 1. Then A = DAD, 


where D = diag(d), dp,... ,d,) with 


Bi Bs +++ Box-1 q BoBa--> Bok 
Se er eS Pk. = Bas 
B2Ba +++ Bok—2 Bi Bs +++ Bok-1 


2k 


Let B = Tat max{6j,1/f;}. Then B-'I < D? < BI, and Fact 2 and Fact 5 apply. Now if all 
l—e <6; <1+e,then(l—e)"' <p '<fB<(1+e)"". 


2. Let A= SHS with S = diag(1, 10, 10”, 10°), and 


a | 1 107! 
*s 1 10? 102 _ 107! 1 107! 
7 102 104 104] ’ 7 10! 31. 107} 


10* 10° 107! 1 
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Suppose that each entry Aj; of A is perturbed to A;;(1+-6;;) with |6;;| < ¢. Then |(AH);;| < ¢|Hij| 
and thus || AH ||. < 1.2e. Since || H~!|| < 10/8, Fact 3 implies 

c(Al,A!) < 1.5e//1— 15¢ © 1.5e. 


15.7 Relative Perturbation Theory for Singular Value Problems 


Definitions: 
Be C”*" is multiplicatively perturbed to BifB= D* BD, for some D, € C”*” and Dz € C"™". 


Denote the singular values of B and B as 
Sv(B) = {o, 02, tee » Omin{mn}}> Sv(B) = {01,0, tee »Omin{mn}}- 


B is said to be (highly) graded if it can be scaled as B = GS such that G is “well-behaved” (i-e., k2(G) is 
of modest magnitude), where S is a scaling matrix, often diagonal but not required so for the facts below. 
Interesting cases are when k2(G)  k2(B). 


Facts: 
1. Let B, B= DBD, € C”*", where D, and Dx are nonsingular. 
om 
(a) [E195] For 1 < j <n, ~—~___ «€; < 9; || Dal Dall. 
PS ODA Ose ee ee 


(b) [Li98], [LM99] 


I|diag ($(01,01),--- 56 (Ons Fn)) ll 
1 1 
< xe — Dot + 5 lle — Dy llu- 


2. [Li99a] Let B, B= Di BD, € C”*" (m > n) have decompositions 


‘i B, 0 UP) ahs 3s 
BI Vl=|'9 pI BIV: Va] = 


Tr 
2; U; 


where [Uj U2]; [Vi V2], [U; U2], and [Vi Va] are unitary, and Uj, Ui eck v,, Vi € C"™** Set 
Oy = O(U,, U)), Oy = O(V,, Vi). TESV( By) 1) SVext(B2) = , then 


V/IIsin @u 22 + [sin Oy |? 
1 7 

SF [Wd — DUI + — DOU 
2 


+0 — DMI +d -— DMI]? 


where 72 = min Q2({U, {Z) over all uw € Sv(By) and jt € SNinlBs ). 

3. [Li98], [Li99a], [LM99] Let B = GS and B = GS be two m x n matrices, where rank(G) = n, 
and let AG = G — G. Then B = DB, where D = I + (AG)G". Fact 1 and Fact 2 apply with 
D, = D and D, = I. Note that 


1 ) I(AG)G "lu 


Drapes (1 
eS C= ACG 2 
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Examples: 


1. [BD90], [DK90], [EI95] B is a real bidiagonal matrix with diagonal entries a, a2,... , a, and off- 
diagonal (the one above the diagonal) entries are bj, b2,... ,by—1. B is the same as B, except for its 
diagonal entries, which change to a4), 024,... ,;@,4,, and its off-diagonal entries, which change 
to B,b1, Bobo, ... » Bn—1bn—1. Then B = D* BD, with 


A AA A) A72A3 
D, = dia (« ; ; ie) 
EB Bibs 
Dy = diag (1,2, 2%, ). 
QQ, AA? 


Leta = ITj-1 max{a@j,1/oj} and p = eS max{6;,1/f;}. Then 


= aX a3 —l 
(@B)~* < (IDo" all Dy lla) < IDillall Dells < os, 


and Fact 1 and Fact 2 apply. Now if all 1 — e < a, Bj < 1+, then (1 — ¢)*""' < (af)! < 
(6) < (1+). 
2. Consider block partitioned matrices 


By Fact 2, €(0j,0;) < $||By' Biall2. Interesting cases are when || By’ Ball is tiny enough to be 
treated as zero and so Sv(B) approximates SV(B) well. This situation occurs in computing the SVD 
of a bidiagonal matrix. 


Author Note: Supported in part by the National Science Foundation under Grant No. DMS-0510664. 
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Eigenvalues often provide great insight into the behavior of matrices, precisely explaining, for example, the 
asymptotic character of functions of matrices like A‘ and e'@. Yet many important applications produce 
matrices whose behavior cannot be explained by eigenvalues alone. In such circumstances further informa- 
tion can be gleaned from broader sets in the complex plane, such as the numerical range (see Chapter 18), 
the polynomial numerical hull [Nev93], [Gre02], and the subject of this section, pseudospectra. 

The ¢-pseudospectrum is a subset of the complex plane that always includes the spectrum, but can 
potentially contain points far from any eigenvalue. Unlike the spectrum, pseudospectra vary with choice 
of norm and, thus, for a given application one must take care to work in a physically appropriate norm. 
Unless otherwise noted, throughout this chapter we assume that A € C”*” is a square matrix with complex 
entries, and that || - || denotes a vector space norm and the matrix norm it induces. When speaking of a 
norm associated with an inner product, we presume that adjoints and normal and unitary matrices are 
defined with respect to that inner product. All computational examples given here use the 2-norm. 

For further details about theoretical aspects of this subject and the application of pseudospectra to a 
variety of problems see [TE05]; for applications in control theory, see [HP05]; and for applications in 
perturbation theory see [CCF96]. 


16.1 Fundamentals of Pseudospectra 


Definitions: 


The e-pseudospectrum of a matrix A € C"*", € > 0, is the set 
o,(A) = {z€C:z€o0(A+ E) forsome E € C"*" with || E|| < ¢}. 


(This definition is sometimes written with a weak inequality, || E || < ¢; for matrices the difference has 
little significance, but the strict inequality proves to be convenient for infinite-dimensional operators.) 

If || Av — zv|| < e|l|v|| for some v 0, then z is an e-pseudoeigenvalue of A with corresponding 
€-pseudoeigenvector v. 

The resolvent of the matrix A € C”*" at a point z ¢ a(A) is the matrix (zI — A)~!. 
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Facts: [TE05] 
1. Equivalent definitions. The set o,(A) can be equivalently defined as: 


(a) The subset of the complex plane bounded within the 1/¢ level set of the norm of the resolvent: 
o,(A) = {z€C: |\(zI — A) |] > e7}}, (16.1) 


with the convention that ||(zI — A)~'|]| = co when zI — Ais not invertible, i.e., when z € o(A). 


(b) The set of all e-pseudoeigenvalues of A: 
o,(A) = {z €C: || Av — zv|| < e for some unit vector v € C"}. 


2. For finite ¢ > 0, o,(A) is a bounded open set in C containing no more than n connected compo- 
nents, and 0(A) C o,(A). Each connected component must contain at least one eigenvalue of A. 
3. Pseudospectral mapping theorems. 


(a) Foranya,y € Cwithy 40, o.(al + yA) =a + o¢/,(A). 


(b) [Lui03] Suppose f is a function analytic on o,(A) for some ¢ > 0, and define 
yle) = SUP} EI << Il f(A+E)— f(A)||. Then f(o,(A)) © 0,(¢)(f (A)). See [Lui03] for several 
more inclusions of this type. 


4. Stability of pseudospectra. For any ¢ > 0 and E such that || E|| < e, 


Oe—E(A) © of, A+E) © Oeqyey (A). 


5. Properties of pseudospectra as ¢ > 0. 


(a) If A is a eigenvalue of A with index k, then there exist constants d and C such that 
|(zI — A)7 "|| < Clz— A 7* for all z such that |z — A| < d. 


(b) Any two matrices with the same ¢-pseudospectra for all ¢ > 0 have the same minimal poly- 
nomial. 


6. Suppose || - || is the natural norm in an inner product space. 


(a) The matrix A is normal (see Section 7.2) if and only if 0,(A) equals the union of open é-balls 
about each eigenvalue for all ¢ > 0. 


(b) For any A € C"*", o,(A*) = o¢(A). 


7. [BLO03] Suppose || - || is the natural norm in an inner product space. The point z = x + iy, 
x,y € R, is on the boundary of o, (A) provided iy is an eigenvalue of the Hamiltonian matrix 


xI — A* el 
-el A-xlI|- 
This fact implies that the boundary of o,(A) cannot contain a segment of any vertical line or, 
substituting e!® A for A, a segment of any straight line. 


8. The following results provide lower and upper bounds on the e-pseudospectrum; A; denotes the 
open unit ball of radius 5 in C, and «(X) = || X||||X7']]. 


(a) Foralle > 0,0(A) + A, Co, (A). 


(b) For any nonsingular S € C”*" and alle > 0, 


Osfx(s (SAS!) © og(A) © ex s)(SAS™"). 


(c) (Bauer—Fike Theorems [BF60], [Dem97]) Let || - || denote a monotone norm. If A is diagonal- 
izable, A= VAV7!, then for all ¢ > 0, 


0-(A) Cc o(A) + Nex(v): 
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(d) 
(e) 


(f) 


(g) 


If A € C"*" has n distinct eigenvalues A,,... ,A,, then for all ¢ > 0, 
o,(A) Cc UA; + Nenk(a;))> 


where « (A; ) here denotes the eigenvalue condition number of A; (i.e., «(Aj ) = 1/ IVivj |, where 
vj; and v; are unit-length left and right eigenvectors of A corresponding to the eigenvalue d j). 


If || - || is the natural norm in an inner product space, then for any ¢ > 0,0¢(A) C W(A)+ Ag, 
where W(-) denotes the numerical range (Chapter 18). 


If || - || is the natural norm in an inner product space and U is a unitary matrix, then 
o,(U* AU) = 0, (A) forall e > 0. 
If || - || is unitarily invariant, then o,(A) C o(A) + Agidepca), where dep(-) denotes Henrici’s 


departure from normality (i.e., the norm of the off-diagonal part of the triangular factor in a 
Schur decomposition, minimized over all such decompositions). 


(Gersgorin Theorem for pseudospectra [ET01]) Using the induced matrix 2-norm, for any 
é> 0, 


0,(A) S Uj (jj aE A,, tea)» 


where rj; = years |ajxl.- 


9. The following results bound o,(A) by pseudospectra of smaller matrices. Here || - || is the natural 
norm in an inner product space. 


(a) 


(c) 


(d) 


[GLO2] If A has the block-triangular form 


then 
o-(A) © o5(B) Uos(E), 


where 5 = (€ + || D|])\/1 + ||Cl|/(e + || Dl). Provided ¢ > || D]l, 
o,(B)Uo,(E) © o,(A), 


where y = € — ||D]|. 

If the columns of V € C"*” form a basis for an invariant subspace of A, and Ve c™ js 
such that V*V = I, then o,(V*AV) C o,(A). In particular, if the columns of U form an 
orthonormal basis for an invariant subspace of A, then o,(U*AU) C o,(A). 

[ET0O1] If U € C”*™ has orthonormal columns and AU = UH + R, thena(H) € o,(A) for 
é = |RI]. 

(Arnoldi factorization) If AU = [U u]H, where H € C+" is an upper Hessenberg matrix 
(hjx = Oif j > k +1) and the columns of [U u] € c"*"+) are orthonormal, then o,(H) C 
o,(A). (The ¢-pseudospectrum of a rectangular matrix is defined in section 16.5 below.) 


Examples: 

The plots of pseudospectra that follow show the boundary of o,(A) for various values of ¢, with the 
smallest values of ¢ corresponding to those boundaries closest to the eigenvalues. In all cases, || - || is the 
2-norm. 


1. The following three matrices all have the same eigenvalues, 0 (A) = {1, +i}, yet their pseudospectra, 
shown in Figure 16.1, differ considerably: 


0 —1 10 0 —1 10 2 —5 10 
1 0 5], 1 0 Sil, 1 -—2 51 
0 0 1 0 0 1 0 0 1 
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1 1 © 
) 0 C) 
+ iy © 
-2 -2 -2 
i 0 1 2 ad 0 1 2 a 0 1 2 


FIGURE 16.1 Spectra (solid dots) and e-pseudospectra of the three matrices of Example 1, each with 
o(A) = {1,i,—i}; ¢ = 107!, 107}, 1077. 


2. [RT92] For any matrix that is zero everywhere except the first superdiagonal, o, (A) consists of an 
open disk centered at zero whose radius depends on ¢ for alle > 0. Figure 16.2 shows pseudospectra 
for two such examples of dimension n = 50: 


0 1 . 3/(n — 1) 
0 


Though these matrices have the same minimal polynomial, the pseudospectra differ considerably. 
3. It is evident from Figure 16.1 that the components of o,(A) need not be convex. In fact, they 
need not be simply connected; that is, o,(A) can have “holes.” This is illustrated in Figure 16.3 
for the following examples, a circulant (hence, normal) matrix and a defective matrix constructed 


= 
ad 


fo) 
oO 


= 0 1 4 0 1 


FIGURE 16.2 Spectra (solid dots) and ¢-pseudospectra of the matrices in Example 2 for e = 107!, 1077,... , 107°. 
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= 


Oo 
i=) 


-2 


+1 0 1 a -4 -2 0 2 


FIGURE 16.3 Spectra (solid dots) and e-pseudospectra (gray regions) of the matrices in Example 3 for ¢ = .5 (left) 
and e = 10? (right). Both plotted pseudospectra are doubly connected. 


by Demmel [Dem87]: 


0100 0 0 0 
0 010 0 0 0 
0001 0 0 0 —1 —100 —10000 
000 0 1 0 0], 0 —1 —100 
00000 1 0 0 0 —1 
00000 0 41 
E 000 0 0 0 


16.2 Toeplitz Matrices 


Given the rich variety of important applications in which Toeplitz matrices arise, we are fortunate that 
so much is now understood about their spectral properties. Nonnormal Toeplitz matrices are prominent 
examples of matrices whose eigenvalues provide only limited insight into system behavior. The spectra of 
infinite-dimensional Toeplitz matrices are easily characterized, and one would hope to use these results 
to approximate the spectra of more recalcitrant large, finite-dimensional examples. For generic problems, 
the spectra of finite-dimensional Toeplitz matrices do not converge to the spectrum of the corresponding 
infinite-dimensional Toeplitz operator. However, the ¢-pseudospectra do converge in the n — oo limit for 
alle > 0, and, moreover, for banded Toeplitz matrices this convergence is especially striking as the resolvent 
grows exponentially with n in certain regions. Comprehensive references addressing the pseudospectra of 
Toeplitz matrices include the books [BS99] and [BG05]. For a generalization of these results to “twisted 
Toeplitz matrices,” where the entries on each diagonal are samples of a smoothly varying function, 
see [TC04]. 


Definitions: 
A Toeplitz operator is a singly infinite matrix with constant entries on each diagonal: 


ag a_-, aA. a_3 


for dy, d41,442,...EC. 
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Provided it is well defined for all z on the unit circle T in the complex plane, the function 
a(z) = 7eo_4, axz* is called the symbol of T. 

The set a(T) C C is called the symbol curve. 

Given a symbol a, the corresponding n-dimensional Toeplitz matrix takes the form 


ag a_;} A2 **+ Aj\~-n 
ay, ag a_| 
Tr=| a. a a ead HS Crs: 
a_\ 
An-1 -'° a2 a\ a0 


For a symbol a(z) = S072, a,z*, the set {T,}n>1 is called a family of Toeplitz matrices. 
A family of Toeplitz matrices with symbol a is banded if there exists some m > 0 such that a4, = 0 for 
allk > m. 


Facts: 


1. [B6ét94] (Convergence of pseudospectra) Let || - || denote any p norm. Ifthe symbol a is a continuous 
function on T, then 


lim o,(T,,) > o¢(T) 


as n —> oo, where T is the infinite-dimensional Toeplitz operator with symbol a acting on the space 
>», and its e-pseudospectrum is a natural generalization of the first definition in section 16.1. The 
convergence of sets is understood in the Hausdorff sense [Hau91, p. 167], i-e., the distance between 
bounded sets X1, Xz C C is given by 


A(2, 2) = mn} sp inf |s} —s,|, sup inf janath 


s;Ed] s2€X2 52€ Dy SIE 


2. [BS99] Provided the symbol a is a continuous function on T, the spectrum o(T) of the infinite 
dimensional Toeplitz operator T on £, comprises a(T) together with all points z € C \ a(T) that 
a(T) encloses with winding number 

1 1 
201i Ja(T) 6 —Z 


dt 


nonzero. From the previous fact, we deduce that ||(zI — T,)~'|| > coasn > oo ifz € o(T) and 
that, for any fixed ¢ > 0, there exists some N > 1 such that o(T) C o,(T,) for alln > N. 

3. [RT92] (Exponential growth of the resolvent) If the family of Toeplitz matrices T,, is banded, then 
for any fixed z € C such that the winding number of a(T) with respect to z is nonzero, there exists 
some y > land N > 1 such that ||(zI — T,,)~'|| > y" foralln > N. 


Examples: 


1. Consider the family of Toeplitz matrices with symbol 


a(t)=t—-5-—G01. 
For any dimension n, the spectrum o (T,,(a)) is contained in the line segment [-$ - 5 i,— 5 + 5 i] in 
the complex plane. This symbol was selected so that o (A) falls in both the left half-plane and the unit 
disk, while even for relatively small values of ¢, o,(A) contains both points in the right half-plane 
and points outside the unit disk for all but the smallest values of n; see Figure 16.4 for n = 50. 
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1.5 


I 
D 
1 
I 
-1.5 L n L 1 
-2 -1.5 -1 -0.5 0 0.5 1 1.5 


FIGURE 16.4 Spectrum (solid dots, so close they appear to be a thick line segment with real part —1/2) and 
€-pseudospectra of the Toeplitz matrix Tso from Example 1; ¢« = 10-!, 10-3, ... , 107~!>. The dashed lines show the 
unit circle and the imaginary axis. 


2. Pseudospectra of matrices with the symbols 


a(t) =it*' +e? +2¢4+5t7? +it? 


and 
4 


a(t) = 3it*+t4+0¢°'43it7 


are shown in Figure 16.5. 


10 


FIGURE 16.5 Spectra (solid dots) and e-pseudospectra of Toeplitz matrices from Example 2 with the first symbol 
on the left (1 = 100) and the second symbol on the right (n = 200), both with e = 10°, 10~,..., 1078. In each plot, 
the gray region is the spectrum of the underlying infinite dimensional Toeplitz operator. 
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FIGURE 16.6 Spectra (solid dots) and e-pseudospectra of Toeplitz matrices for the discretization of a convection— 
diffusion operator described in Application 1 with v = 1/50 and three values of n; ¢ = 107', 10-2, ..., 10~°. The 
gray dots and lines in each plot show eigenvalues and pseudospectra of the differential operator to which the matrix 
spectra and pseudospectra converge. 


Applications: 


1. [RT94] Discretization of the one-dimensional convection-diffusion equation 
vu" (x) + u(x) = f(x), u(0)=u(1)=0 


for x € [0,1] with second-order centered finite differences on a uniform grid with spacing 
h = 1/(n + 1) between grid points results in an n x n Toeplitz matrix with symbol 


B= vi 1 , 2v v 1 wi 
. =(4+5) (F)+(4 =) 


On the right-most part of the spectrum, both the eigenvalues and pseudospectra of the discretization 
matrix converge to those of the underlying differential operator 


Lu=vu"+u' 


whose domain is the space of functions that are square-integrable over [0, 1] and satisfy the boundary 
conditions u(0) = u(1) = 0; see Figure 16.6. 


16.3. Behavior of Functions of Matrices 


In practice, pseudospectra are most often used to investigate the behavior of a function of a matrix. Does 
the solution x(t) = e'4x(0) or x, = Axo of the linear dynamical system x’(t) = Ax(t) or x,4) = Ax, 
grow or decay as t,k —> oo? Eigenvalues provide an answer: If o(A) lies in the open unit disk or left 
half-plane, the solution must eventually decay. However, the results described in this section show that if 
€-pseudoeigenvalues of A extend well beyond the unit disk or left half-plane for small values of ¢, then the 
system must exhibit transient growth for some initial states. While such growth is notable even for purely 
linear problems, it should spark special caution when observed for a dynamical system that arises from 
the linearization of a nonlinear system about a steady state based on the assumption that disturbances 
from that state are small in magnitude. This reasoning has been applied extensively in recent years in fluid 
dynamics; see, e.g., [TTRD93]. 


Definitions: 


The €-pseudospectral abscissa of A measures the rightmost extent of o,(A): a (A) = sup,c,.(4) Rez. 
The €-pseudospectral radius of A measures the maximum magnitude in o, (A): p,(A) = SUP,¢9,(A) |ZI- 
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Facts: [TE05, §§14-19] 


1. For € > 0, supyepyso lle’ |] = ae(A)/e. 


10. 


11. 


For € > 0, suppentso | A* |] > (pe(A) — 1)/e. 
For any function f that is analytic on the spectrum of A, 


I f(A eel: 


Equality holds when || - || is the natural norm in an inner product space in which A is normal. 
In the special case of matrix exponentials e’4 and matrix powers A*, this last fact implies that 


jje“A > e'@, LAR = p(A)* 


for all t > 0 and integers k > 0, where a(A) = maxjeq(4) Red is the spectral abscissa and 
p(A) = max;<o a) |A| is the spectral radius. 

Let I’, be a finite union of Jordan curves containing o,(A) in their collective interior for some 
€ > 0, and suppose f is a function analytic on I’, and its interior. Then 


€ 


I f(ADI < 


max | f (z)|, 

are maxi fe) 

where L, denotes the arc-length of T',. 

In the special case of matrix exponentials e’4 and matrix powers A‘, this last fact implies that for 
all t > 0 and integers k > 0, 


L 
Ay s set, AT < p(A) Ye. 
2 
In typical cases, larger values of ¢ give superior bounds for small t and k, while smaller values of ¢ 
yield more descriptive bounds for larger t and k; see Figure 16.7. 
Suppose z € C\o(A) witha = Rezandeé = 1/||(zI — A)~!||. Provided a > e¢, then for any fixed 
tT > 0, 


Ta 


ae 
A 

sup |le’"|| = a 

0<t<t at e(e = 1) 


Suppose z € C \o(A) witha = Rezande = 1/||(zI — A)~'||, and that ||e*4|| < M forallt > 0 
with M > a/e. Then for any t > 0, 


Ile] = e!(1—eM/a) + eM/a. 


Suppose z € C \ o(A) withr = |z| and e = 1/||(zI — A)~!||. Provided r > 1+ ¢, then for any 
fixed integer k > 1, 


r“(r —l—e)+er*7} 
sup ||A*|| > —. 
O0<k<K« (r —1—e)+er* 


Suppose z € C \ o(A) withr = |z| and e = 1/||(zI — A)~!||, and that || A*|| < M for all integers 
k > 0 with M > (r — 1)/e. Then for any integer k > 0, 


A‘ > rk (7 —1 —eM) + eM. 


For any A € C"*", 


(A) —1 
A‘ < e(k4 Nees as 
e>0 éE 
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12. (Kreiss Matrix Theorem) For any A € C”™", 


sup BANE < sup ||A*|| < en sup plAy= 1 
e>0 € k>0 e>0 é 
13. [GT93] There exist matrices A and B such that, in the induced 2-norm, o,(A) = o,(B) for all 
€ > 0, yet || f(A)|lo ¥ || f(B)|l2 for some polynomial f; see Example 2. That is, even if the 2-norm 
of the resolvents of A and B are identical for all z € C, the norms of other matrix functions in A 
and B need not agree. (Curiously, if the Frobenius norm of the resolvents of A and B agree for all 
z €C, then || f(A) |p = || f(B)||e for all polynomials f.) 


I|A‘] 


0 50 100 150 200 


FIGURE 16.7 The functions ||e’4|| and || A*|| exhibit transient growth before exponential decay for the Toeplitz 
matrix of dimension n = 50, whose pseudospectra were illustrated in Figure 16.4. The horizontal dashed lines show 
the lower bounds on maximal growth given in Facts 1 and 2, while the lower dashed lines show the lower bounds of 
Fact 4. The gray lines show the upper bounds in Fact 6 fore = 107!, 10-7, ... , 10-78 (ordered by decreasing slope). 


Examples: 


1. Consider the tridiagonal Toeplitz matrix of dimension n = 50 from Example 1 of the last section, 
whose pseudospectra were illustrated in Figure 16.4. Since all the eigenvalues of this matrix are 
contained in both the left half-plane and the unit disk, e’4 + 0.ast > oo and Ak > Oask > ov. 
However, 0, (A) extends far into the right half-plane and beyond the unit disk even for ¢ as small 
as 10-”. Consequently, the lower bounds in Facts 1 and 2 guarantee that ||e“|| and || A‘ || exhibit 
transient growth before their eventual decay; results such as Fact 6 limit the extent of the transient 
growth. These bounds are illustrated in Figure 16.7. (For a similar example involving a different 
matrix, see the “Transient” demonstration in [Wri02b].) 

2. [GT93] The matrices 


0 
0 
OiNecs 2 
J2 
0 


> 

II 
oe 6.6 9°98 
ooo OF 
oooro 
OS? 6. 26. "ES 
ooosc Oo 


0 
1 
0 
0 
0 


ee 
oocUCcOlUCUCOCOULUCUCOO 
<a > aa <a > ee 


have the same 2-norm ¢-pseudospectra for all ¢ > 0. However, || All, = /2 > 1 = || Bll2. 


Pseudospectra 16-11 


Applications: 


1. Fact 5 leads to a convergence result for the GMRES algorithm (Chapter 41), which constructs 
estimates x; to the solution x of the linear system Ax = b. The kth residual r, = b — Ax; is 


bounded by 
Ilrx ll2 L. ; 
< min max|p(2)| 
Ilroll2 2E peClzsk] 240 
p(0)=1 


where C[z; k] denotes the set of polynomials of degree k or less, I’, is a finite union of Jordan curves 
containing o,(A) in their collective interior for some ¢ > 0, and L, is the arc-length of T',. 

2. For further examples of the use of pseudospectra to analyze matrix iterations and the stability of 
discretizations of differential equations, see [TE05, §§24—34]. 


16.4 Computation of Pseudospectra 


This section describes techniques for computing and approximating pseudospectra, focusing primarily 
on the induced matrix 2-norm, the case most studied in the literature and for which very satisfactory 
algorithms exist. For further details, see [Tre99], [TE05, §§39-44], or [Wri02a]. 


Facts: [TE05] 


1. There are two general approaches to computing pseudospectra, both based on the expression for 
o,(A) in Fact 1(a) of Section 16.1. The most widely-used method computes the resolvent norm, 
||(zI— A)! Ip, ona grid of points in the complex plane and submits the results to a contour-plotting 
program; the second approach uses a curve-tracing algorithm to track the e~'-level curve of the 
resolvent norm ([Brii96]). Both approaches exploit the fact that the 2-norm of the resolvent is the 
reciprocal of the minimum singular value of zI — A. A third approach, based on the characterization 
of o,(A) as the set of all e-pseudoeigenvalues, approximates o,(A) by the union of the eigenvalues 
of A+ E for randomly generated E € C”*" with ||E|| < e. 

2. For dense matrices A, the computation of the minimum singular value of zI — A requires O(n?) 
floating point operations for each distinct value of z. Hence, the contour-plotting approach to 
computing pseudospectra based on a grid of m x m points in the complex plane, implemented via 
the most naive method, requires O(m?n*) operations. 

3. [Lui97] Improved efficiency is obtained through the use of iterative methods for computing the 
minimum singular value of the resolvent. The most effective methods (inverse iteration or the inverse 
Lanczos method) require matrix-vector products of the form (zI — A)~'x at each iteration. For 
dense A, this approach requires O(n*) operations per grid point. One can decrease this labor to 
O(n’) by first reducing A to Schur form, A = UTU*, and then noting that ||(zI — A)“!|]2 = 
|(zI — T)~!||2. Vectors of the form (zI — T)~!x can be computed in O(n’) operations since 
T is triangular. As the inverse iteration and inverse Lanczos methods typically converge to the 
minimum singular value in a small number of iterations at each grid point, the total complexity of 
the contour-plotting approach is O(n? + mn’). 

4. For large-scale problems (say, n > 1000), the cost of preliminary triangularization can be pro- 
hibitive. Several alternatives are available: Use sparse direct or iterative methods to compute 
(zI — A)~'xat each grid point, or reduce the dimension of the problem by replacing A with a smaller 
matrix, such as the (m-+ 1) x m upper Hessenberg matrix in an Arnoldi decomposition, or U* AU, 
where the columns of U € C”*™ form an orthonormal basis for an invariant subspace correspond- 
ing to physically relevant eigenvalues, with m « n. As per results stated in Fact 9 of Section 16.1, 
the pseudospectra of these smaller matrices provide a lower bounds on the pseudospectra of A. 
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5. [Wri02b] EigToolis a freely available MATLAB package based ona highly-efficient, robust implemen- 
tation of the grid-based method with preliminary triangularization and inverse Lanczos iteration. 
For large-scale problems, EigTool uses ARPACK (Chapter 76), to compute a subspace that includes 
an invariant subspace associated with eigenvalues in a given region of the complex plane. The 
EigTool software, which was used to compute the pseudospectra shown throughout this section, 
can be downloaded fromhttp: //www.comlab.ox.ac.uk/pseudospectra/eigtool. 

6. Curve-tracing algorithms can also benefit from iterative computation of the resolvent norm, though 
the standard implementation requires both left and right singular vectors associated with the min- 
imal singular value ([Brii96]). Robust implementations require measures to ensure that all com- 
ponents of o,(A) have been located and to handle cusps in the boundary; see, e.g., [BG01]. 

7. Software for computing 2-norm pseudospectra can be used to compute pseudospectra in any norm 
induced by an inner product. Suppose the inner product of x and y is given by (x,y) w = (Wx, y), 
where (-,-) denotes the Euclidean inner product and W = L L*, where L* denotes the conjugate 
transpose of L. Then the W-norm pseudospectra of A are equal to the 2-norm pseudospectra of 
L*AL™. 

8. Fornorms not associated with inner products, all known grid-based algorithms require O(n?) oper- 
ations per grid point, typically involving the construction of the resolvent (zI — A)~!. Higham and 
Tisseur ([HT00]) have proposed an efficient approach for approximating 1-norm pseudospectra 
using a norm estimator. 

9. [BLO03],[MO05] There exist efficient algorithms, based on Fact 7 of section 16.1, for computing 
the 2-norm pseudospectral radius and abscissa without first determining the entire pseudo- 
spectrum. 


16.5 Extensions 


The previous sections address the standard formulation of the e-pseudospectrum, the union of all eigen- 
values of A + E for a square matrix A and general complex perturbations E, with ||E || < ¢. Natural 
modifications restrict the structure of E or adapt the definition to more general eigenvalue problems. The 
former topic has attracted considerable attention in the control theory literature and is presented in detail 
in [HPO5]. 


Definitions: 


The spectral value set, or structured ¢-pseudospectrum, of the matrix triplet (A, B,C), A € C”*", 
Bec"™*™, C € C*", for ¢ > 0 is the set 


o0,(A; B,C) = {z€C:z€oa0(A+ BEC) forsome E € C”*? with || E|| < e}. 
The real structured ¢-pseudospectrum of A € R"*” is the set 
of(A) = {ze o(A+E): E €R™",||E|| < e}. 
The e-pseudospectrum of a rectangular matrix A € C’*” (n > m) for € > 0 is the set 
o,(A) = {ze C: (A+ E)x = zx for some x $ 0 and || E|| < e}. 


[Ruh95] For A € C’*" and invertible B € C”*", the e-pseudospectrum of the matrix pencil A — 1B 
(or generalized eigenvalue problem Ax = Bx) for ¢ > 0 is the set 


o.(A, B) = 0,(B7!A). 
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[THO1] The ¢-pseudospectrum of the matrix polynomial P(A) (or polynomial eigenvalue problem 
P(A)x = 0), where P(A) = AP Ap +A? Ap_1 +--+ + Ao and € > 0, is the set 
o,(P)={zeC:zeoa(P+E) for some 
E(A) =APE,+---+ Eo, ||Ejll < eaj;, j =0,..., ph 
for values a,... ,@p. For most applications, one would either take aw; = 1 for all j, or aj = || Aj||. (This 


definition differs considerably from the one given for the pseudospectrum of a matrix pencil. In particular, 
when p = | the present definition does not reduce to the above definition for the pencil; see Fact 6 below.) 


Facts: 
1. [HP92, HK93] The above definition of the spectral value set o,(A; B, C) is equivalent to 
o,(A; B,C) = {z € C: ||C(zI — A)! BI] > e7}}. 
2. [Kar03] The above definition of the real structured ¢-pseudospectrum oR(A) is equivalent to 


o®(A) = {z€ C:r(A,z) < e}, 


€ 


where 


r(A,z)'= inf o 


Re(zI — A)“! —yIm(zI — A)7! 
ye(0,1) : 


y—lIm (zI — A)7! Re (zI — A)~ 


and o>(-) denotes the second largest singular value. From this formulation, one can derive algorithms 
for computing oR(A) akin to those used for computing 0, (A). 

3. The definition of o%(A) suggests similar formulations that impose different restrictions upon E, 
such as a sparsity pattern, Toeplitz structure, nonnegativity or stochasticity of A + E, etc. Such 
structured pseudospectra are often difficult to compute or approximate. 

4. [WT02] The above definition of the ¢-pseudospectrum o, (A) of a rectangular matrix A € C”*”, 
n > m, is equivalent to 


o-(A) = {z EC: (zl — A)t | > e 4}, 


where (-)' denotes the Moore—Penrose pseudoinverse and T denotes the n x m matrix that has the 
m xX m identity in the first m rows and is zero elsewhere. 
5. The following facts apply to the ¢-pseudospectrum of a rectangular matrix A € C”*", m > n. 


(a) [WT02] It is possible that o,(A) = 9. 


(b) [BLO04] For A € C"*",m > n,andanye > 0, the seto,(A) contains no more than 2n?—n+ 1 
connected components. 


6. [TE05] Alternative definitions have been proposed for the pseudospectrum of the matrix pencil 
A — AB. The definition presented above has the advantage that the pseudospectrum is invariant 
to premultiplication of the pencil by a nonsingular matrix, which is consistent with the fact that 
premultiplication of the differential equation Bx’ = Ax does not affect the solution x. Here are 
two alternative definitions, neither of which are equivalent to the previous definition. 


(a) [Rie94] If B is Hermitian positive definite with Cholesky factorization B = LL*, then the 
pseudospectrum of the pencil can be defined in terms of the standard pseudospectrum of a 
transformed problem: 


o.(A, B) = 0,(L~'AL~*). 
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FIGURE 16.8 Spectrum (solid dot) and real structured e¢-pseudospectra o®(A) (left) and unstructured 
é-pseudospectra a, (A) of the second matrix of Example 3 in section 16.1 for ¢ = 1077, 107‘. 


(b) [FGNT96, TH01] The following definition is more appropriate for the study of eigenvalue 
perturbations: 


o,(A, B)={zeC: (A+ Eo)x = z(B + E,)x for some 
x # Oand Eo, E, with || Eo|| < ea, || Eil] < eo}, 


where generally either aj = 1 for j = 0,1, or a = || Al| and a = ||B||. This is a special case 
of the definition given above for the pseudospectrum of a matrix polynomial. 


7. [THO1] The above definition of the ¢-pseudospectrum ofa matrix polynomial, o, (P), is equivalent 


to 
o.(P) ={z€C: ||P(z)" I] > 1/(eg(lzl))}, 


where #(z) = SB, a,.z* for the same values of a,... 5a p used in the earlier definition. 


2 2 2 

1 1 1 

0 0 0 
-1 -1 -1 
2 -2 -2 

-1 0 1 2 -1 0 1 2 -1 0 1 2 


FIGURE 16.9 _ ¢-pseudospectra of the rectangular matrix in Example 2 with 6 = 0.02 (left), 6 = 0.01 (middle), 
5 = 0.005 (right), and ¢ = 107!, 107!°, and 1077. Note that in the first two plots, o,(A) = @ fore = 10°?. 
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Examples: 


1. Figure 16.8 compares real structured ¢-pseudospectra o8(A) to the (unstructured) pseudospectra 
o,(A) for the second matrix in Example 3 of Section 16.1; cf. [TE05, Fig. 50.3]. 
2. Figure 16.9 shows pseudospectra of the rectangular matrix 


a5! G0 
1-2. 5i 

A= , 
OF 00° 3 
5 6 6 


which is the third matrix in Example 1 of Section 16.1, but with an extra row appended. 
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17.1 Definitions and Characterizations 


Singular values and the singular value decomposition are defined in Chapter 5.6. Additional information 
on computation of the singular value decomposition can be found in Chapter 45. A brief history of the 
singular value decomposition and early references can be found in [HJ91, Chap. 3]. 

Throughout this chapter, q = min{m, n}, and if A € C”*" has real eigenvalues, then they are ordered 
Ai (A) 2 +++ 2 An(A). 


Definitions: 


For A € C”*", define the singular value vector sv(A) = (0)(A),... ,04(A)). 

For A € C”*", define r}(A) > +++ > fm(A) and c;(A) > --- > c,(A) to be the ordered Euclidean 
row and column lengths of A, that is, the square roots of the ordered diagonal entries of AA* 
and A*A. 

For A € C”*" define | A| pq = (A* A)/?, This is called the spectral absolute value of A. (This is also 
called the absolute value, but the latter term will not be used in this chapter due to potential confusion 
with the entry-wise absolute value of A, denoted | A|.) 

A polar decomposition or polar form of the matrix A € C”*” with m > n isa factorization A = UP, 
where P € C"™” is positive semidefinite and U € C”™” satisfies U*U = I,. 
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Facts: 


The following facts can be found in most books on matrix theory, for example [HJ91, Chap. 3] or 
[Bha97]. 


1. Take A e C”™", and set 


AO 
0 Of. 


Then 0;(A) = 0;(B) fori = 1,...,q and o;(B) = 0 fori > q. We may choose the zero blocks 
in B to ensure that B is square. In this way we can often generalize results on the singular values 
of square matrices to rectangular matrices. For simplicity of exposition, in this chapter we will 
sometimes state a result for square matrices rather than the more general result for rectangular 
matrices. 

. (Unitary invariance) Take A ¢ C”*” . Then for any unitary U € C”*” and V € C"*", 


N 


0;(A) =0;(UAV), 4152500259. 


3. Take A, B ¢ C”*". There are unitary matrices U € C’”*” and V € C”*" such that A = UBV if 
and only ifo;(A) = 0;(B),7 = 1,2,...,q. 


4, Let Ae C”*". Then o?(A) = Aj(AA*) = A;(A* A) fori = 1,2,...5q. 
5. Let Ae C”*". Let S; denote the set of subspaces of C” of dimension i. Then fori = 1,2,... ,q, 
o;(A)= min max || Ax||, = min max || Ax|l2, 
XESy-i41 XEX,||x|]2=1 YeSj-1 x Ly,||x\]2=1 
o;(A) = max min || Ax||, = max min || Axl. 
XeS; xeX,||x\|.=1 VESn-j xLy,||xl]2=1 


6. Let A € C”*"and define the Hermitian matrix 


A 
J E Cmnmtn | 
A* 0 
The eigenvalues of J are to)(A),... ,+o,(A) together with |m — n| zeros. The matrix J is called 


the Jordan—Wielandt matrix. Its use allows one to deduce singular value results from results for 
eigenvalues of Hermitian matrices. 

7. Take m > nand A € C”™*". Let A = UP bea polar decomposition of A. Then 0;(A) = A;(P), 
tel La ara? fe 

8. Let Ae C”*" and 1 <k < q. Then 


k 
S9)(A)Smax(Re tr OAV IU eC Vel US VV Si) 
i=l 

k 

[][ oi(A4) = max{idetu* av] :U ee C"™ ,VieC™, UU = VV = i}. 


i=1 


If m = n, then 


S$ 9;(A) = p> \(U*AU), |; VEC™, UU = s} 
i=1 


i=1 


We cannot replace the n by a general k € {1,... ,n}. 
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9. Let Ae C”*”. A yields 
(a) o;(AT) = 0;(A*) = 0;(A) = 0;(A), fori = 1,2,...,q. 


(b) Let k = rank(A). Then o;(Al) = 4-441 (A) fori = 1,...,k, ando;(A') = 0 fori = 
k+1,...,q. In particular, ifm =n and A is invertible, then 


o)(A!) = GA); b= yee ns 


(c) For any 7 €N 


o((A*A))) =07/(A), i=1,...5q3 


t 


oj((A*A)i A*) = 0;(A(A*A)/) =o7*1(A) i= 1,...,¢. 


10. Let UP bea polar decomposition of A € C”*” (m > n). The positive semidefinite factor P is 
uniquely determined and is equal to | A| pq. The factor U is uniquely determined if A has rank n. If 
A has singular value decomposition A = U; DU; (U; € C”*", Ur € C”*"), then P = U,XUZ, 
and U may be taken to be U; US. 

11. Take A,U € C”*" with U unitary. Then A = U|A|pq if and only if A = | A*|,qU. 


Examples: 
1. Take 

I 8 5)! 4 

Los Sa: Ar 

25... Vie ts 3 

=3 Ii I) =5 


The singular value decomposition of A is A= UXV*, where & = diag(20, 12, 8,4), and 


—l 1 -1 1 —l 1 -1 1 

1}-1 -l 1 1 1 af 1 1 1 
U=-— and V=-— 

2 1 -1l -1 1 2 1 -1 -1 1 

1 1 1 1 -1 -l 1 1 


The singular values of A are 20, 12, 8, 4. Let Q denote the permutation matrix that takes (x), x2, x3, x4) 
to (x1, X4, x3, X2). Let P = |Alpq = QA. The polar decomposition of A is A = QP. (To see this, 
note that a permutation matrix is unitary and that P is positive definite by Gerschgorin’s theorem.) 
Note also that | Al pa 4 |A*|pa = AQ. 


17.2 Singular Values of Special Matrices 


In this section, we present some matrices where the singular values (or some of the singular values) are 
known, and facts about the singular values of certain structured matrices. 


Facts: 


The following results can be obtained by straightforward computations if no specific reference is given. 


1. Let D = diag(a),... ,a@,,), where the a; are integers, and let H, and H) be Hadamard matrices. 
(See Chapter 32.2.) Then the matrix H; DH) has integer entries and has integer singular values 
nlay|,. o. , ||. 
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(2 x 2 matrix) Take A € C?*?. Set D = | det(A)|?, N = || All%. The singular values of A are 
N+ JSN*—4D 
5 . 
Let X € C”*" have singular values 0) > --- > o4 (q = min{m, n}). Set 


I 2x 
0 I 


A = € Cmtnmtn 


The m+ n singular values of A are 


Gal oy Plas hag yee Ff Wilasagl af OPP leper op ples 


[HJ91, Theorem 4.2.15] Let A € C™*™ and B € C"*”™ have rank mand n. The nonzero singular 
values of A ® B areo;(A)oj(B),i = 1,...,m,j =1,...,n. 

Let A € C”*” be normal with eigenvalues A,,... ,A,, and let p be a polynomial. Then the singular 
values of p(A) are | p(Ax)|,k = 1,... ,n.In particular, if A isa circulant with first row do,... ,4n—1, 
then A has singular values 


n—-1 
S aye 27tiK/n any co — a eee 7 
j=0 


Take A € C”*” and nonzero x € C”. If Ax = Ax and x*A = Ax*, then |A| is a singular value of A. 
In particular, if A is doubly stochastic, then o;(A) = 1. 

[Kit95] Let A be the companion matrix corresponding to the monic polynomial p(t) = t” + 
a,—\t” | +---+ayt+ap.SetN=14+ Nees. |a;|?. The n singular values of A are 


ye N?=4lao?? pe N2 — 4laol 
2) + Roa? ey Sat 2 2 . 


[Hig96, p. 167] Take s,c € R such that s? + c? = 1. The matrix 


1 -c -c¢ —C 
1 -c —Cc 
A= diag(1,s,... ys) 
—c 
1 


is called a Kahan matrix. If c and s are positive, then o,_;(A) =s" 7/1 +c. 
[GE95, Lemma 3.1] Take 0 = dj < dy <--- < d, and0 4 z; € C. Let 


Z dy 


Zn dy, 


The singular values of A satisfy the equation 
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and exactly one lies in each of the intervals (d), d2),... 5 (dn—1, dn), (dns dn + ||Z||2). Leto; = 0; (A). 
The left and right ith singular vectors of A are u/||u||2 and v/||v||2 respectively, where 


Z1 Zn I d l dyZ2 dnZn 

u= a. ¥ an v= Ras 
2 >a? a 
i dy — 0; di — 9; 


T 


10. (Bidiagonal) Take 
B= ec", 


If all the a; and f; are nonzero, then B is called an unreduced bidiagonal matrix and 
(a) The singular values of B are distinct. 
(b) The singular values of B depend only on the moduli of ay, ...,@,, Bi, ..-5 Bn—1. 


(c) The largest singular value of B is a strictly increasing function of the modulus of each of the 


a; and §;. 


(d) The smallest singular value of B is a strictly increasing function of the modulus of each of the 
a; and a strictly decreasing function of the modulus of each of the £;. 


(e) (High relative accuracy) Take t > 1 and multiply one of the entries of B by t to give B. Then 
t~10;(B) < o;(B) < t9;(B). 


11. [HJ85, Sec. 4.4, prob. 26] Let A € C”*” be skew-symmetric (and possibly complex). The nonzero 
singular values of A occur in pairs. 


17.3. Unitarily Invariant Norms 


Throughout this section, q = min{m, n}. 


Definitions: 


A vector norm || - || on C”*” is unitarily invariant (w.i.) if || Al] = ||UAV]| for any unitary U € C”*” 
and V € C”*" andany A € C”™". 

| - llur is used to denote a general unitarily invariant norm. 

A function g:R" > Rt is a permutation invariant absolute norm if it is a norm, and in addition 
2(X1,--- Xn) = Z([xi|,--- 5|Xn|) and g(x) = g(Px) for all x € R” and all permutation matrices P € 
R"*". (Many authors call a permutation invariant absolute norm a symmetric gauge function.) 

The Ky Fan k norms of A € C”*" are 


k 
Allke= So ofA), RH 12..059 


i=1 


The Schatten-p norms of A € C’”*" are 


q 1/p 
|Allsp = (sora) = (tr|Ale,)” 0< p< 


i=1 


Il All soo = 01 (A). 
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The trace norm of A € C’*” is 


q 
[|All = > 0;(A) = [Allg = l|Allsa = tr |Alpa. 


i=1 


Other norms discussed in this section, such as the spectral norm || - || (|| Allz = 01(A) = maxxzo Lele ) 


and the Frobenius norm || - |p (/Allzp = (S04, 07(A))!? = (C7, Pea |a;;|°)'/*), are defined in 
Section 7.1. and discussed extensively in Chapter 37. 


Warning: There is potential for considerable confusion. For example, || Al|2 = || Allx,1 = ||Alls,oo, while 
ll - loo F Il + Ilsjoo (unless m = 1), and generally || Alla, || Alls,2 and || Allx,2 are all different, as are || All, 
|| Alls, and || Allx,1. Nevertheless, many authors use || - || for |] - || x,4 and || - |lp for | - Ils,p- 

Facts: 


The following standard facts can be found in many texts, e.g., [HJ91, §3.5] and [Bha97, Chap. IV]. 


1. Let || - || bea norm on C”%". It is unitarily invariant if and only if there is a permutation invariant 
absolute norm g on R? such that || Al] = g(oi(A),... ,0q(A)) for all Ae C"*". 

2. Let ||-|| bea unitarily invariant norm on C”””, and let g be the corresponding permutation invariant 
absolute norm g. Then the dual norms (see Chapter 37) satisfy || Al]? = g?(o1(A),... »0q(A)). 

3. [HJ91, Prob. 3.5.18] The spectral norm and trace norm are duals, while the Frobenius norm is self 
dual. The dual of || - |ls,p is || - Ils,p, where 1/p + 1/p = land 


[| Allir 
VAI = max { Al pop? Kah 


4, Forany Ae C™*",q7? | Alle < Allo < IlAlle- 

5. If || - || isa u.i. norm on C”*", then N(A) = || A* All’? is a ui. norm on C"*". A norm that arises 
in this way is called a Q-norm. 

6. Let A, B € C”” be given. The following are equivalent 


(a) ||Allur < || Blluz for all unitarily invariant norms || - || vr. 
(b) |Allke < WBllxx fork = 1,2,...,q. 
(c) (0)(A),... 504(A)) Xw (o1(B),... ,04(B)). (Kw is defined in Preliminaries) 


The equivalence of the first two conditions is Fan’s Dominance Theorem. 
7. The Ky—Fan-k norms can be represented in terms of an extremal problem involving the spectral 
norm and the trace norm. Take A € C’”*”. Then 


Alle = min{|| Xl + kVllp:X+Y = A} k=1,...,4. 


8. [HJ91, Theorem 3.3.14] Take A, B €¢ C”*". Then 


q 
|trAB*| < S$ °0;(A)o;(B). 


i=1 


This is an important result in developing the theory of unitarily invariant norms. 
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Examples: 
1. The matrix A in Example 1 of Section 17.1 has singular values 20, 12, 8, and 4. So 


|Allz = 20, |Alle = V624, [Aller = 445 
Allk, = 20, ||Allx.2 = 32, ||Allx,3 = 40, ||Allk = 445 
Alls. =44, |All. = 624, |IAlls,3 = 710304 = 21.7605, || Allsoo = 20. 


17.4 Inequalities 


Throughout this section, q = min{m,n} and if A € C”™” has real eigenvalues, then they are ordered 
A(A) = +++ = An(A). 


Definitions: 


Pinching is defined recursively. If 


Ai 0 


= Cre 
> 
Ax 


then B is a pinching of A. (Note that we do not require the Aj; to be square.) Furthermore, any pinching 
of B is a pinching of A. 
For positive a, 8, define the measure of relative separation x(a, 8) = |./a/B — /8/a|. 


Facts: 


The following facts can be found in standard references, for example [HJ91, Chap. 3], unless another 
reference is given. 


1. (Submatrices) Take A € C”™" and let B denote A with one of its rows or columns deleted. Then 
Oj41(A) < 0;(B) <o;(A), i=1,...,q—1. 
2. Take A € C”*" and let B be A with a row and a column deleted. Then 


0j42(A) < o;(B) <o;)(A), i=1,...,q-2. 


The i + 2 cannot be replaced by i + 1. (Example 2) 
3. Take A € C”*" and let B be an (m — k) x (n — 1) submatrix of A. Then 


Gi+e4(A) So)(B) <o,(A), t=1L...5q—(k +l). 


4. Take A € C”*" and let B be A with some of its rows and/or columns set to zero. Then o;(B) < 
0;(A), 1=1,...,q. 

5. Let B bea pinching of A. Then sv(B) <,, sv(A). The inequalities Tes o;(B) < Nea o;(A) and 
0;.(B) < o,(A) are not necessarily true for k > 1. (Example 1) 

6. (Singular values of A+ B) Let A, Be C”*". 


(a) sv(A+ B) X, sv(A) + sv(B), or equivalently 


k k k 
So oi(A+ B) < So oi(A) +> 9;(B), i =1,...,9. 
i=1 i=1 i=l 


(b) Ifi + j —1 <q andi,j €N, thenoj4;_)(A + B) < o;(A) +0;(B). 
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(c) We have the weak majorization |sv(A + B) — sv(A)| <w sv(B) or, equivalently, if 1 < i, < 
+ < iz <q, then 


k k 
S > |o;,(A + B) — 0;,(A)| < 5° 0; (B), 


j=l j=l 


k k k k k 
3 0;,(A) — $5 9j(B) < $5 0;,(A + B) < 55 0;,(A) + 5 0j(B) 
i=l j=l j=l i=l j=l 


(d) [Tho75] (Thompson’s Standard Additive Inequalities) If 1 < i, <--- <i, <q,1 <i, < 
-<ip Sqandi,+ jp <q +k, then 
k 


k k 
do Gi.+j-s(A+ B) <> 0;,(A) + 97 93, (B). 
s=l s=l s=l1 


7. (Singular values of AB) Take A,B € C"*". 
(a) Forallk = 1,2,...,nandall p > 0, we have 


i=n—k+1 i=n—k+1 


I] o(Aoi(B) < J] (AB), 


i=n i=n 


k k 

[[o(48) < [[oi(Aoi(8 
i=l i=l 

k k 

S- af (AB) < S-a?(A)o?(B) 
i=l i=l 


(b) Ifi, 7 e Nandi + j —1 <n, then oj4;-,(AB) < o;(A)o;(B). 
(c) on(A)o;(B) < o;(AB) < 0)(A)o;(B),i = 1,2,...,n 
(d) [LM99] Take 1 < j,) <--- < jx <n. If Ais invertible and oj,(B) > 0, then o;,(AB) > 0 and 
e 0; (AB) o;(B) 
o;(A) < ax{ bi 7 \. o;(A). 
I [I Oj, (B)? o;,(AB) I 


i=n—k+1 


(e) [LM99] Take invertible S,T €¢ C"”". Set A = SAT. Let the singular values of A and Abe 
0, >-+: >o, and 6; >--- > 6,. Then 


: - : 1 = = 
I|diag(X (01,61), 5--- > X(On Gn) )llur < 5 (|S* — Slur + IT* — Tur) - 


(f) [TT73] (Thompson’s Standard Multiplicative Inequalities) Take 1 < i, <--- < im < nand 
1 <j <+++ < jm <1. Mim + jm < m+n, then 


[[oi+i-s(4B) < []o,(4) [] 9, (8). 
s=l s=l s=l 
8. [Bha97, §IX.1] Take A, B € C”™”. 
(a) If AB is normal, then 
k 


k 
[[o48) = [To (BA), k=1,...,9 


and, consequently, sv(AB) <,, sv(B A), and || AB||ur < ||BAllur. 
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(b) If AB is Hermitian, then sv(AB) =<, sv(H(BA)) and ||ABl|y; < || H(BA)|lur, where 
H(X) = (X 4+ X*)/2. 


9. (Term-wise singular value inequalities) [Zha02, p. 28] Take A, B € C”*”. Then 
20;(AB*) < o;(A*A+ B*B), i=1,...,q 


and, more generally, if p, fp > Oand 1/p + 1/p = 1, then 


A*A)P/2 — (B*B)P/2 Al? Bi? 
a(aB") <a (| 45 ) =o (Ses led , 


P P P 


The inequalities 20,(A*B) < o(A*A + B*B) ando,(A+ B) < 0 (|Alpa + |Blpa) are not true 
in general (Example 3), but we do have 


|A* Bila, < WA*Allur ll B*Bllur. 
10. [Bha97, Prop. II.5.1] Take A € C"*". Then A;(A + A*) < 20;(A),i = 1,2,...,n. 


R 0O 
| € C"*" (R € C?*?) have singular values 


11. [LM02] (Block triangular matrices) Let A = i T 


a) >-++>a,. Letk = min{p,n — p}. Then 


(a) Ifomin(R) > Omax(T), then 


o;(R) 
a; < oj-)(T), PS pae lyn he 


lA 


aj, PS isp 


(b) (o1(S),..- ,0K(S)) Xw (1 = Otny +++ 5 Ok — An—K+1): 


(c) If A is invertible, then 


1 : ~ 
(o,(T7'S),... ,o.(T 'S)) <i, (2 & ee Ak a 1) 


2 \Qy Qa) Qn—k+1 Ak 
ele . . . Au Ai “* a: 
12. [LM02] (Block positive semidefinite matrices) Let A = Ae A € C"*" be positive definite 
12 22 


with eigenvalues 4; > +++ > A,. Assume Ay; € C?*?. Set k = min{p,n — p}. Then 


j j 
[][o7(4n) < [J oiAwoi(An) 7 =1...5k 
i=1 


i=1 


(01(Ag?Ara)>+- .0k(An Anz) Sw (Wit — Vases VR = Van) » 


_ a 1 
(o (Ay A), sae ,0%(Aj, Ai)) yw 2 (x(A1. An), see X (Ak An—k41)) . 
If k = n/2, then 
Avllar < WAullur || Azllur- 


13. (Singular values and eigenvalues) Let A € C"*". Assume |A;(A)| > +++ > |An(A)|. Then 
(a) [TK |Ai(A)| < Ti, oi(A), k= 1,...,n, with equality for k =n. 
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(b) Fix p > 0. Then for k = 1,2,...,n, 


k k 
> AP(A)] < So of (A). 
i=l i=l 


Equality holds with k = n if and only if equality holds for all k = 1,2,... ,n, if and only if A 
is normal. 


(c) [HJ91, p. 180] (Yamamoto’s theorem) limp5o0(0;(A*))!/F = |a;(A)|, i= 1,...,n. 


14. [LMO01] Leta; € Cando; € Ro, i = 1,...,n be ordered in nonincreasing absolute value. There 
is a matrix A with eigenvalues A,... ,4, and singular values o),... , 0, if and only if 


k k 
[[ ail < Il: k=1,...,n, with equality for k =n. 
i=l i=l 


In addition: 


(a) The matrix A can be taken to be upper triangular with the eigenvalues on the diagonal in any 
order. 


(b) Ifthe complex entries in Aj,... , A, occur in conjugate pairs, then A may be taken to be in real 
Schur form, with the 1 x 1 and 2 x 2 blocks on the diagonal in any order. 


(c) There is a finite construction of the upper triangular matrix in cases (a) and (b). 
(d) Ifm > 2, then A cannot always be taken to be bidiagonal. (Example 5) 

15. [Zha02, Chap. 2] (Singular values of Ao B) Take A, B € C"*". 
(a) o;(Ao B) < min{r;(A),c;(B)}-o,(B), i=1,2,...,n. 


(b) We have the following weak majorizations: 


k k 
S > 0;(Ao B) < S> min{r;(A),c;(A)}o)(B),  k=1,...,n, 


i=l i=1 


k k 
di oi(Ao B) < Sl o;(A)oi(B), k=1,...50, 
i=1 i=1 


k k 
[[o7(40B) s [J oi((4*4) 0 (B*B)), k=1,...,n. 


i=l i=1 


(c) Take X,Y € C"*". If A = X*Y, then we have the weak majorization 
k 


k 
S 5 0;(Ao B)< > ¢j(X)ei(Y)oi(B), k=1,...,n. 


i=l i=l 
(d) If B is positive semidefinite with diagonal entries bj, > --- > Dy, then 


k k 
S"a;(Ao B)< > byoj(A), k=1,... 50. 
i=1 i=1 


(e) Ifboth A and B are positive definite, then so is A o B (Schur product theorem). In this case 
the singular values of A, B and A o B are their eigenvalues and B A has positive eigenvalues 
and we have the weak multiplicative majorizations 


[]atpoaita) < [[biata) = [p(ea) < [piace B), k= 15 2yatc phe 
i=k i=k i=k i=k 


The inequalities are still valid if we replace A o B by Ao B?. (Note B® is not necessarily the 
same as B* = B.) 
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16. 


Le 


18. 


19. 


20. 


21. 


22. 
23. 


Let A € C”*". The following are equivalent: 


(a) o)(Ao B) <o;(B) forall Be C”™". 


(b) SK o;(Ao B) < SE, (B) for all B € C"*" andallk =1,... ,q. 


(c) There are positive semidefinite P € C"*" and Q € C”*” such that 


PB iA 
A* Q 


is positive semidefinite, and has diagonal entries at most 1. 


(Singular values and matrix entries) Take A € C’*". Then 
rf 


(lanl, lar’... l@iinl*) as (07 (A)... ,0,(A),0,... ,0), 
q 
of (A) — 


1 1 


A 
ibs 

2 
=) 
A 
s 
lA 
A 


1 


If o,(A) = |a;;|, then all the other entries in rowi and column j of A are 0. 
Take o, >--->0, > Oanda, >--- >a, > 0. Then 


JA ER" st. o)(A)=0; and c)(A)=a; © (pares 5a.) < Gene ir) 


This statement is still true if we replace R’*” by C"*" and/or c;(- ) by ri(- ). 
Take A € C”*". Then 


The case k = 1 is Hadamard’s Inequality: | det(A)| < []j_, ¢;(A). 

[Tho77] Take F = C or Rand d,...,d, € F such that |d,| > --- > |d,|,ando, >--->o, > 0. 
There is a matrix A € F"*" with diagonal entries d),... ,d, and singular values o,... ,o, if and 
only if 


n-1 n—-1 
(di|,.-+51dul) Sw (01(A),.-.,0n(A)) and © |djl — Idnl < $5 oj(A) — 9A). 
j=l j=l 


(Nonnegative matrices) Take A = [ajj] € Ce, 
(a) If B = [|a;;|], then o,(A) < o;(B). 


(b) If A and B are real and 0 < aj; < bj; V i,j, then o,(A) < o;(B). The condition 0 < aj; is 
essential. (Example 4) 


(c) The condition 0 < bj; < 1 V i,j does not imply o,(Ao B) < o;(A). (Example 4) 


(Bound on o,) Let A € C”*". Then || A]lz = 0)(A) < JAN Allo: 
[Zha99] (Cartesian decomposition) Let C = A+iB € C"*", where A and B are Hermitian. Let 
A, B,C have singular values aj, Bj, yi, j = 1,...,n. Then 


(Yb-- . Yn) x<w J2(lary + iB,|,. oe Qn + iBy|) Xw 2("3- os >Vn)« 
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Examples: 
1. Take 
11 1 1 0 0 1 0 0 
A=/1 1 1], B=]0 1 1], C= ]0 1 0 
Bi: do 0 1 1 0 0 1 


Then B isa pinching of A, and C isa pinching of both A and B. The matrices A, B, C have singular 
values a = (3,0,0), 8 = (2,1,0), and y = (1,1, 1). As stated in Fact 5, y xy B <y a. In fact, 
since the matrices are all positive semidefinite, we may replace <,, by <. However, it is not true 
that y; < a; except fori = 1. Nor is it true that | det(C)| < | det(A)|. 


2. The matrices 


ll —3 —5 1 
ll —3 —5 1 ll -—3 —5 

1 —-5 -3 11 
A= », B= 1-5 -3 Ill}, C= lL. 5° 3 
=o 1 11 —3 =) 1 ll 


25. ik < tae 3 
=3 (qi) =| 


have singular values w = (20, 12, 8,4), 8 = (17.9, 10.5, 6.0), and y = (16.7, 6.2, 4.5) (to 1 decimal 
place). The singular values of B interlace those of A (a4 < B3 < a3 < By < a < By, < a), but 
those of C do not. In particular, a3 ¢ yp. It is true that aj4. < y; < a; (i = 1,2). 


3. Take 
1 0 0 1 
A= | and B= | : 
1 0 0 1 


Then ||A-+ Blly = 0(A+ B) = 2 £ V2 = o4(|Alpa + [Blpa) = Il |Alpa + [Blpa lo. Also, 
20\(A*B) = 4 42=0,(A*A+ B*B). 
4. Setting entries of a matrix to zero can increase the largest singular value. Take 


1 1 1 1 
=| | and =| | 
-1 1 0 1 


Then o;(A) = V2 < (1+ /5)/2 = 01(B). 

5. A bidiagonal matrix B cannot have eigenvalues 1, 1, 1 and singular values 1/2, 1/2, 4. If B is 
unreduced bidiagonal, then it cannot have repeated singular values. (See Fact 10, section 17.2.) 
However, if B were reduced, then it would have a singular value equal to 1. 


17.5 Matrix Approximation 


Recall that || - ||y denotes a general unitarily invariant norm, and that q = min{m, n}. 


Facts: 


The following facts can be found in standard references, for example, [HJ91, Chap. 3], unless another 
reference is given. 


1. (Best rank k approximation.) Let A € C”™*" and1 < k < q—1.Let A= UX V* bea singular value 
decomposition of A. Let © be equal to © except that 4;; = 0 fori > k, andlet A= US V*. Then 
rank(A) < k, and 


|= — Llur = A — Allur = min{|| A — Bllur : rank(B) < k}. 
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In particular, for the spectral norm and the Frobenius norm, we have 


Oc+1(A) = min{||A — B]l2 : rank(B) < k}, 


j 1/2 
( ys ot) 


i=k+1 


min{||A — B||r : rank(B) < ky}. 


2. [Bha97, p. 276] (Best unitary approximation) Take A, W € C"*" with W unitary. Let A = UP bea 
polar decomposition of A. Then 


|A— Ullur < ||A— Wlluor < ||A+ Ulor. 
3. [GV96, §12.4.1] [HJ85, Ex. 7.4.8] (Orthogonal Procrustes problem) Let A, B € C”*". Let B* A have 
a polar decomposition B* A = UP. Then 
||A— BUllp = min{|A — BWlp :WeEC™, Wwe. 
This result is not true if || - || ¢ is replaced by || - ||uz ({[Mat93, §4]). 


4. [Hig89] (Best PSD approximation) Take A € C"*". Set An = (A+ A*)/2, B = (An + |An|)/2). 
Then B is positive semidefinite and is the unique solution to 


min{|| A — X||p : X € C"*”, X € PSD}. 


There is also a formula for the best PSD approximation in the spectral norm. 
5. Let A,B € C”*” have singular value decompositions A = U,u,Vx and B = UgugV5z. Let 
Ue C”*” and V € C”*" be any unitary matrices. Then 


24 — Lallur < ||A— UBV" |lur. 


17.6 Characterization of the Eigenvalues of Sums 
of Hermitian Matrices and Singular Values of Sums 
and Products of General Matrices 


There are necessary and sufficient conditions for three sets of numbers to be the eigenvalues of Hermitian 
A,B,C = A+B € C"*", or the singular values of A,B,C = A+ B € C”*", or the singular values 
of nonsingular A, B, C = AB € C"*". The key results in this section were first proved by Klyachko 
([Kly98]) and Knutson and Tao ([KT99]). The results presented here are from a survey by Fulton [Ful00]. 
Bhatia has written an expository paper on the subject ([Bha01]). 


Definitions: 


The inequalities are in terms of the sets 7,” of triples (I, J, K ) of subsets of {1,... ,n} of the same cardinality 
r, defined by the following inductive procedure. Set 


Tie i=Dere+vph, 


iel jo keK 


U" = {ane 


When r = 1, set T;’ = UY. In general, 
S {ne € U! |for all p <r and all 


(F,G,H) in TL, oie t+ Sig < Sokn + ppt vl. 


feF geG heH 


In this section, the vectors a, 6, y will have real entries ordered in nonincreasing order. 
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Facts: 
The following facts are in [Ful00]: 


1. A triple (a, 8, y) of real n-vectors occurs as eigenvalues of Hermitian A,B,C = A+ B eC" if 
and only if 3° y; = 0 a; + ¥ A; and the inequalities 
dom = a+ dB; 
keK iel joJ 
hold for every (I, J, K) in T,”, for allr < n. Furthermore, the statement is true if C"*” is replaced 
by R?*". 
2. Take Hermitian A,B € C”*” (not necessarily PSD). Let the vectors of eigenvalues of A, B, 
C = A+B bea, B, and y. Then we have the (nonlinear) inequality 


n n n 
minzes, | [(@i + Bra) < [] vi < maxres, | [(ai + Br). 
i=l = i=l 


3. Fixm,n and set q = min{m, n}. For any subset X of {1,... , m+n}, define X, = {i : i € X,i < q} 
and xO = {i:i<q, m+n+1-i € X}.A triple (a, f,y) occurs as the singular values of 
A,B,C = A+B eC", ifand only if the inequalities 


Sou - Sov < Soa - Soa, + 4B) - 5OB; 


keKy keK/ iel iel iq jel) 


are satisfied for all (I, J, K) in T/"*", for allr < m+n. This statement is not true ifC”*” is replaced 
by R”*". (See Example 1.) 

4. A triple of positive real n-vectors (a, 6, y) occurs as the singular values of n by n matrices A,B, 
C = AB €C"" ifand only if y, «++ Y_) = 1 +++ On B+ ++ By and 


I[v <I] -][4 


keK iel joJ 


for all (1, J, K) in T,”, and allr < n. This statement is still true if C”*” is replaced by R"*". 


Example: 


1. There are A,B,C = A+B € C?*? with singular values (1, 1), (1,0), and (1, 1), but there are no 
A, B,C = A+B € R’”’ with these singular values. 

In the complex case, take A = diag(1, 1/2 + (/3/2)i); B = diag(0,—1). 

Now suppose that A and B are real 2 x 2 matrices such that A and C = A+ B both have 
singular values (1, 1). Then A and C are orthogonal. Consider BC’ = AC’ —CC? = AC? —I. 
Because AC" is real, it has eigenvalues a, @ and so BC T has eigenvalues a — 1, @ — 1. Because AC 7 
is orthogonal, it is normal and, hence, so is BC T and so its singular values are |a@ — 1| and |a — 1], 


which are equal and, in particular, cannot be (1,0). 


17.7 Miscellaneous Results and Generalizations 


Throughout this section F can be taken to be either R or C. 


Definitions: 
Let 4’, Y be subspaces of C” of dimension m and n. The principal angles 0 < 6; < --- < 0, < 1/2 
between ¥ and ¥ and principal vectors u;,... ,u, and vj,... , Vq are defined inductively: 


cos(9,) = max{|x"y| : x € Ames Ixll2 = llyll2 = 1}. 
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Let u, and v, bea pair of maximizing vectors. For k = 2,...,q, 
cos(6) = max{|x"y|: x € ¥,y € Y, ||xllo = llyll2=1, xu; =y*vj =0, i=1,...,k— 1}. 


Let u, and v; be a pair of maximizing vectors. (Principal angles are also called canonical angles, and the 
cosines of the principal angles are called canonical correlations.) 


Facts: 
1. (Principal Angles) Let X, Y be subspaces of C’ of dimension m and n. 
(a) [BG73] The principal vectors obtained by the process above are not necessarily unique, but the 
principal angles are unique (and, hence, independent of the chosen principal vectors). 


(b) Let m = n < r/2 and X,Y be matrices whose columns form orthonormal bases for the 
subspaces ¥ and Y, respectively. 


i. The singular values of X*Y are the cosines of the principal angles between the subspaces V 
and y. 
ii. There are unitary matrices U € C’*” and Vx and Vy € C”*” such that 


In Tr 
UXVx = On, |, UYW= Dn 
O,—nn 0, —nn 


where I’ and & are nonnegative diagonal matrices. Their diagonal entries are the cosines 
and sines respectively of the principal angles between ¥ and J. 


(c) [QZL05] Take m = n. For any permutation invariant absolute norm g on R”, 
g(sin(O)),...,sin(O,)), g(2sin(6,/2),...,2sin(0,,/2)), and g(@,... 56m) 
are metrics on the set of subspaces of dimension n of C’*". 


2. [GV96, Theorem 2.6.2] (CS decomposition) Let W € F”*" be unitary. Take a positive integer | 
such that 2] < n. Then there are unitary matrices Uj,, Vi; € F *! and Un, Vin € FP * such 


that 
Uy 0 Vi (0 er 
- U. ‘. Gam lipe he rule 
22 22 0 Gt wag 


where [ = diag(yi,... , 7) and & = diag(o1,... ,07) are nonnegative and ’?+pr2=I]. 

3. [GV96, Theorem 8.7.4] (Generalized singular value decomposition) Take A € F?*" and B € F™*" 
with p > n. Then there is an invertible X € F”*", unitary U € F?*? and V © F™*”, and 
nonnegative diagonal matrices U4 € R’*" and Xz € R?*4 (q = min{m, n}) suchthat A = UX AX 
and B = Vd,X. 
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The numerical range W(A) of ann x n complex matrix A is the collection of complex numbers of the form 
x" Ax, where x € C” isa unit vector. It can be viewed as a “picture” of A containing useful information of A. 
Even if the matrix A is not known explicitly, the “picture” W(A) would allow one to “see” many properties 
of the matrix. For example, the numerical range can be used to locate eigenvalues, deduce algebraic and 
analytic properties, obtain norm bounds, help find dilations with simple structure, etc. Related to the 
numerical range are the numerical radius of A defined by w(A) = max,ewia) |u| and the distance of 
W(A) to the origin denoted by w(A) = min,ewya) |u|. The quantities w(A) and w(A) are useful in 
studying perturbation, convergence, stability, and approximation problems. 

Note that the spectrum o (A) can be viewed as another useful “picture” of the matrix A € M,. There 
are interesting relations between 0 (A) and W(A). 


18.1 Basic Properties and Examples 


Definitions and Notation: 
Let A € C"*”. The numerical range (also known as the field of values) of A is defined by 


W(A) = {x* Ax: x € C",x*x= 1}. 
The numerical radius of A and the distance of W(A) to the origin are the quantities 
w(A) = max{|u|: 4 € W(A)} and w(A) = min{|py|: € W(A)}. 
Furthermore, let 


W(A) = fa:a€ W(A)}. 


18-1 


18-2 


Facts: 
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The following basic facts can be found in most references on numerical ranges such as [GR96], [Hal82], 
and [HJ91]. 


1. Let Ae C"*", a,b € C. Then W(aA+ bI) =aWw(A)+b. 


SON ST 


10. 


11. 


12. 


Let A € C’™". Then W(U*AU) = W(A) for any unitary U € C”*", 
Let A € C"*". Suppose k € {1,...,n — 1} and X € C"** satisfies X*X = I,. Then 


W(X* AX) C W(A). 


In particular, for any k x k principal submatrix B of A, we have W(B) C W(A). 


Let A € C"*". Then W(A) is a compact convex set in C. 

If A; ® A, € M,,, then W(A) = conv {W(A,) U W(A))}. 

Let A € M,. Then W(A) = W(A‘) and W(A*) = W(A). 

If A € C?** has eigenvalues 41,2, then W(A) is an elliptical disk with foci 4,, 42, and minor axis 

with length {tr (A* A) — |Ai|? — |A2|?}!/?. Consequently, if A = ‘ | , then the minor axis of 
2 


the elliptical disk W(A) has length |b]. 

Let A € C’*". Then W(A) is a subset of a straight line if and only if there are a,b € C witha 4 0 
such that aA + bI is Hermitian. In particular, we have the following: 

(a) A= al ifand only if W(A) = {a}. 

(b) A = A* ifand only if W(A) CR. 

(c) A = A* is positive definite if and only if W(A) C (0,00). 

(d) A = A* is positive semidefinite if and only if W(A) C [0, oo). 

If A € C”*” is normal, then W(A) = convo (A) is a convex polygon. The converse is true ifn < 4. 
Let A € C’*". The following conditions are equivalent. 

(a) W(A) = convo(A). 

(b) W(A) is a convex polygon with vertices j11,..., [k. 

(c) Ais unitarily similar to diag (W11,..., 44) © B such that W(B) C conv {j1),..., Ue}. 

Let A € C"*". Then A is unitary if and only if all eigenvalues of A have modulus one and 
W(A) = convo(A). 

Suppose A = (Ajj)1<i,j<m € My is a block matrix such that Ay,,..., Amm are square matrices and 
Ajj = 0 whenever (i, j) ¢ {(1,2),...,(m— 1,m), (m,1)}. Then W(A) = cW(A) for any c € C 
satisfying c” = 1. If Aj) is also zero, then W(A) is a circular disk centered at 0 with the radius 
equal to the largest eigenvalue of (A + A*)/2. 


Examples: 
1. Let A = diag (1,0). Then W(A) = [0, 1]. 
2. Let A= ; . Then W(A) is the closed unit disk D = {a € C: |a| < 1}. 
3. Let A= 


0 


2 ; : : : 
1 . Then by Fact 7 above, W(A) is the convex set whose boundary is the ellipse with 
foci 1 and 2 and minor axis 2, as shown in Figure 18.1. 


Let A = diag (1,i,-1,-i) ® 


. By Facts 5 and 7, the boundary of W(A) is the square with 


vertices 1,i, —1, —i. 
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FIGURE 18.1 Numerical range of the matrix A in Example 3. 


Applications: 


1. By Fact 6, if A is real, then W(A) is symmetric about the real axis, i.e, W(A) = W(A). 
2. Suppose A € C”*", and there are a,b € C such that (A — aI)(A — bI) = 0,. Then A is unitarily 
similar to a matrix of the form 
a a 
0 b 


with d| > --- > d; > 0, wherer +s + 2t = n. By Facts 1, 5, and 7, the set W(A) is the elliptical 
disk with foci a, b and minor axis of length d, where 


dy 


a 
al, ® bl, ® 0 b QO: @D 


d = d = { (|All — lal?) (WAI — 1b?) } 711 Alla 


ift > 1,andd = 0 otherwise. 

3. By Fact 12, if A € C"*” is the basic circulant matrix Ey. + E23 +--+ + En—in + Eni, then 
W(A) = conv{c € C: c" = 1}; if A € M, is the Jordan block of zero J,,(0), then W(A) = 
{c €C: |c| < cos(z/(n + 1))}. 

4. Suppose A € C”*” is a primitive nonnegative matrix. Then A is permutationally similar to a 
block matrix (Aj) as described in Fact 12 and, thus, W(A) = cW(A) for any c ¢€ C satisfying 
reside —ael Es 


18.2 The Spectrum and Special Boundary Points 


Definitions and Notation: 
Let 0S and int (S) be the boundary and the interior of a convex compact subset S of C. 

A support line @ of S is a line that intersects dS such that S lies entirely within one of the closed 
half-planes determined by €. 

A boundary point yz of S is nondifferentiable if there is more than one support line of S passing 
through ju. 

An eigenvalue A of A € C’*” is a reducing eigenvalue if A is unitarily similar to [A] @ Ao. 


Facts: 
The following facts can be found in [GR96],[Hal82], and [HJ91]. 


1. Let A € C"*". Then 
a(A) © WA) € {a €C: Ja| < | Alla}. 
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2. Let A, E € C”*". We have 


o(A+ E)CW(A+E) C W(A)+ W(E) 
C{at+beC:ae WA), bEC with |b] < JE |}. 


3. Let Ae C"*" anda € C. Thena € o0(A)N 0W(A) if and only if A is unitarily similar to al, ® B 
such that a ¢ o(B) Uint (W(B)). 

4. Let A € C”*" anda € C. Then a is a nondifferentiable boundary point of W(A) if and only if A 
is unitarily similar to aI, © B such that a ¢ W(B). In particular, a is a reducing eigenvalue of A. 

5. Let A € C”*". If W(A) has at least n — 1 nondifferentiable boundary points or if at least n — 1 
eigenvalues of A (counting multiplicities) lie in 0 W(A), then A is normal. 


Examples: 


1. Let A= [1] 6 


; A‘ . Then W(A) is the unit disk centered at the origin, and 1 is a reducing 


eigenvalue of A lying on the boundary of W(A). 


0 


2. Let A = [2] ® 0 


4 . Then W(A) is the convex hull of unit disk centered at the origin of the 


number 2, and 2 is a nondifferentiable boundary point of W(A). 


Applications: 


1. By Fact 1, if A € C"*" and 0 ¢ W(A), then 0 ¢ o(A) and, thus, A is invertible. 

2. By Fact 4, if A €¢ C”*", then W(A) has at most n nondifferentiable boundary points. 

3. While W(A) does not give a very tight containment region for o(A) as shown in the examples 
in the last section. Fact 2 shows that the numerical range can be used to estimated the spectrum 
of the resulting matrix when A is under a perturbation E. In contrast, o(A) and o(E£) usually 


: : i 0 M 
do not carry much information about o(A + E) in general. For example, let A = | and 


0 0 


a : i Then o(A) = o(E) = {0}, 0(A+ E) = {£VMe} © W(A + E), which is the 


elliptical disk with foci + Me and length of minor axis equal to | |M| — |e| |. 


18.3 Location of the Numerical Range 


Facts: 
The following facts can be found in [HJ91]. 


1. Let A € C”*" and t € [0, 277). Suppose x; € C” is a unit eigenvector corresponding to the largest 
eigenvalue A, (t) of e'’ A + e~* A*, and 


P, ={aeC:e"ate “a <A,(t)}. 
Then 


e!W(A) CPi, Ap = xP Ax; € DW(A) NAP, 
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and 
W(A) = Pyefoan) e”P, = conv {A, :r € [0,27)}. 
If T = {th,...,t} withO < tf <--- <  < 2m andk > 2 such that % — t > z, then 
P(A) =Mrere"P, and P(A) =conv{A, :r € T} 
are two polygons in C such that 
P7(A) S W(A) © PP (A). 


Moreover, both the area W(A)\ P(A) and the area of PP (A)\ W(A) converge to 0as max{tj—tj-1: 
1 < j <k+ 1} converges to 0, where f = 0, t+ = 27. 
2. Let A = (aj) € C"*". For each j = 1,...,n, let 


gi= So (ail + |aji|)/2 and G;(A)={a€C:|a—ajj| < gj}. 


ifj 
Then 
W(A) © conv Ui Gj(A). 
Examples: 
1. Let A= ; ; . Then W(A) is the circular disk centered at 2 with radius 1 In Figure 18.2, W(A) 


is approximated by P(A) with T = {2k7/100:0 < k < 99}. If T = {0,2 /2,7, 32/2}, then 


the polygon P.P(A) in Fact 1 is bounded by the four lines {3 + bi : b € R}, {a t+i:a € R}, 
{1+ bi : b € R}, {a—i : a € R}, and the polygon P}(A) equals the convex hull of {2, 1+i,0, 1—i}. 


Si 2 3 
2. LetA=|4 —3i —2].In Figure 18.3, W(A) is approximated by P(A) with T = {2k/100: 
1 3 9 


0 < k < 99}. By Fact 2, W(A) lies in the convex hull of the circles G; = {a € C: |a — 5i| < 5}, 
G.={aeEC:|lat+3i| <5.5},G3 = {a eC: la —9| < 4.5}. 


imaginary axis 


“0.5 1 1.5 2 25 3 3.5 
real axis 


FIGURE 18.2 Numerical range of the matrix A in Example 1. 
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imaginary axis 


“6 4 2 0 2 4 6 8 10 12 
real axis 


FIGURE 18.3. Numerical range of the matrix A in Example 2. 


Applications: 
1. Let A= H +iG, where H,G € C”™*" are Hermitian. Then 


W(A) C W(H) +iW(G) = {a+ib:ae€ W(H), b € W(G)}, 


which is P,P (A) for T = {0,2/2, 2, 32/2}. 
2. Let A = H + iG, where H,G € C"*” are Hermitian. Denote by A)(X) > --- > A,(X) for a 
Hermitian matrix X € C"*". By Fact 1, 


w(A) = max{A,(costH + sintG) : t € [0,27)}. 
If0 ¢ W(A), then 
w(A) = max{{d,,(costH + sintG) : t € [0,27r)} U {O}}. 
3. By Fact 2, if A = (aj) € C"™”, then 
w(A) < max{lajj] +gj:1< j <n}. 


In particular, if A is nonnegative, then w(A) = A,(A + A‘)/2. 


18.4 Numerical Radius 


Definitions: 


Let N bea vector norm on C”*". It is submultiplicative if 
N(AB) < N(A)N(B) forall A,BeC"*". 
It is unitarily invariant if 
N(UAV) = N(A) forall Ae C"*" and unitary U, V € C"*”. 
It is unitary similarity invariant (also known as weakly unitarily invariant) if 


N(U*AU) = N(A) forall Ae C"*" and unitary U € C"*". 
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Facts: 


The following facts can be found in [GR96] and [HJ91]. 


ie 


The numerical radius w(-) is a unitary similarity invariant vector norm on C”*”, and it is not 
ys y 

unitarily invariant. 

For any A € C"*”, we have 


p(A) < w(A) < ||Allz < 2w(A). 


Suppose A € C”*” is nonzero and the minimal polynomial of A has degree m. The following 
conditions are equivalent. 


(a) p(A) = w(A). 

(b) There exists k > 1 such that A is unitarily similar to yU © B for a unitary U € Ckx* and 
Be C’-xC-)) with w(B) < w(A) = y. 

(c) There exists s > m such that w(A‘’) = w(A)*. 

Suppose A € C”*" is nonzero and the minimal polynomial of A has degree m. The following 

conditions are equivalent. 


(a) p(A) = |JAlle. 

(b) w(A) = |[All2. 

(c) There exists k > 1 such that A is unitarily similar to yU © B for a unitary U € Ckx* and a 
Be C"®x-® with || Bll, < ||All2 = y- 

(d) There exists s > m such that || A*||2 = || All5. 

Suppose A € C”*" is nonzero. The following conditions are equivalent. 

(a) ||All2 = 2w(A). 

(b) W(A) is a circular disk centered at origin with radius || A||/2. 


(c) A/||Allz is unitarily similar to A; @ A such that A; = 0 


and w(A2) < 1. 
The vector norm 4w on C”*" is submultiplicative, i.e., 
4w(AB) < (4w(A))(4w(B)) forall A,B eC"*". 


The equality holds if 


Let A € C"*" and k be a positive integer. Then 
w(A*) < w(A)k. 


Let N be a unitary similarity invariant vector norm on C”*” such that N (A‘) < N(A) for any 
A € C"*" and positive integer k. Then 


w(A) < N(A) forall AeC"*”. 


Suppose N is a unitarily invariant vector norm on C”*". Let 


7 2Tk B Ox ifn = 2k, 
~ )2h@ OO ifn=2k4+1. 
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Then a = N(E,,) and b = N(D) are the best (largest and smallest) constants such that 


aw(A) < N(A) < bw(A) forall AEC”. 


10. Let A € C”*". The following are equivalent: 


(a) w(A) <1. 
(b) A, (eA + e7* A*)/2 < 1 forall t € [0, 277). 
(c) There is Z € C"*” such that ers I . Zz is positive semidefinite. 


(d) There exists X € C*”*" satisfying X*X = I, and 


18.5 Products of Matrices 


Facts: 
The following facts can be found in [GR96] and [HJ91]. 


1. Let A, B € C"*" be such that w(A) > 0. Then 
o(A7'B) C {b/a:a € W(A),b € W(B)}. 


2. LetO<t <t <t) +m and S = {re :r > 0, t € [t,h]}. Theno(A) C S ifand only if there is 
a positive definite B € C”*” such that W(AB) C S. 
3. Let A, B E C"™". 


(a) If AB = BA, then w(AB) < 2w(A)w(B). 
(b) If.A or B is normal such that AB = BA, then w(AB) < w(A)w(B). 
(c) If A* = aI and AB = BA, then w(AB) < || All2w(B). 
(d) If AB = BA and AB* = B* A, then w(AB) < min{w(A)]|| Bll.) || Allw(B)}. 
4, Let Aand B be square matrices such that A or B is normal. Then 
W(Ao B) C W(A® B) = conv {W(A)W(B)}. 
Consequently, 


w(Ao B) < w(A® B) = w(A)w(B). 


(See Chapter 8.5 and 10.4 for the definitions of t Ao Band A ® B.) 
5. Let A and B be square matrices. Then 


w(Ao B) < w(A@ B) < min{w(A)||Bll2, || Allaw(B)} < 2w(A)w(B). 
6. Let A € C”*". Then 
w(Ao X)<w(X) forall Xe Cc"*" 


if and only if A = B*WB such that W satisfies || W|| < 1 and all diagonal entries of B*B are 
bounded by 1. 
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Examples: 


1. Let A € C?*? be the Jordan block of zero Jo(0), and B = A? + A’. Then w(A) = w(B) = 
cos(z/10) < 1 and w(AB) = 1 > ||Al|2w(B). So, even if AB = BA, we may not have w(AB) < 
min{w(A)|| Bll2, || Allaw(B)}. 


1 


mre eo 
0 


| W(A) = {a€C:|a—1| < 1/2} and 


W(A*) = {ae C:|a—1]<, 
whereas 
conv W(A)* C {se’ € C: 5 € [0.25,2.25], t € [—1/3,2/3]}. 
So, W(A’) Z conv W(A)’. 
3. LetA = 


1 
W(B) = W(A)W(B) = [—1,1]. So, o(AB) Z conv W(A)W(B). 


: _ aad a= | j|-Theno(AB) = 6), W(AB) = lb a}and WUA) = 


Applications: 


1. If C € C"*" is positive definite, then W(C7') = W(C)7! = {c7! : c € W(C)}. Applying Fact 1 
with A = C7!, 


o (CB) C W(C)W(B). 


2. If C € C"*" satisfies w(C) > 0, then for every unit vector x € C"” x*C7!x = y*C*C7!Cy with 
y = C_!x and, hence, 


W(C7!) C{rb:r>0, be W(C*)}={rb:r>0, be W(C)}. 
Applying this observation and Fact 1 with A = C~!, we have 


o(AB) C{rab:r>0, aeWw(A), be W(B)}. 


18.6 Dilations and Norm Estimation 


Definitions: 


A matrix A € C”*" has a dilation B € C”*” if there is X € C”™*" such that X*X = I, and X*BX = A. 
A matrix A € C"*" is a contraction if || Al|2 < 1. 


Facts: 
The following facts can be found in [CL00],[CLO1] and their references. 
1. Ahasa dilation B if and only if B is unitarily similar to a matrix of the form 


A x 
es a 


2. Suppose B € C>*? hasa reducing eigenvalue, or B € C?*?_If W(A) C W(B), then A has a dilation 
of the form B @® I. 
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3. Letr € [—1,1]. Suppose A € C"*” is a contraction with 
W(A) CS = {a€C:a+4 < 2r}. 


Then A has a unitary dilation U € C?"*?" such that W(U) C S. 
4. Let A € C”™*". Then 


W(A) = M{W(B) : B € C*"*" is a normal dilation of A}. 
If A is a contraction, then 
W(A) = N{W(U) : U € C*"*?" is a unitary dilation of A}. 


5. Let Ae C"*", 


(a) If W(A) lies in an triangle with vertices z), Z2,z3, then 


| Alla S max{|Z1], |Z21, |Z3]}- 


(b) If W(A) lies in an ellipse € with foci A), A2, and minor axis of length b, then 


Allo < (Vaal + [Aal)? +b? + Vaal — [A2|)? + b?}/2. 


More generally, if W(A) lies in the convex hull of the ellipse € and the point zo, then 


[Alle < max {Izols (Val + lal)? + 8? + Val = Daal)? + 9/2} 
6. Let A € C”*". Suppose there is t € [0,2z7) such that e!'W(A) lies in a rectangle R centered at 


Zo € C with vertices z +a +ifB and z +a + if, where a, B > 0, so that z} = z) +a + if has 
the largest magnitude. Then 


|All < IZ1| ifR c conv {Z1,Z1,—Z1}, 
saad ees B otherwise. 


The bound in each case is attainable. 


Examples: 


1. Let A= 


So Oe 
| 

a 

° 


0 O -i 


Then W(A) C W(B). However, A does not have a dilation of the form B ® I, for either of the 
matrices because 


Allo = V2 > 1 = [Blo = |B @ Inllz. 


So, there is no hope to further extend Fact 1 in this section to arbitrary B € C?*3 or normal matrix 
Bec**4, 
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18.7 Mappings on Matrices 


Definitions: 


Let @ : C"*" > C”*"™ be a linear map. It is unital if d(I,,) = I; it is positive if @(A) is positive 
semidefinite whenever A is positive semidefinite. 


Facts: 


The following facts can be found in [GR96] unless another reference is given. 


1 


[HJ91] Let P(C) be the set of subsets of C. Suppose a function F : C"*" —+ P(C) satisfies the 
following three conditions. 


(a) F(A) is compact and convex for every A € C”*”. 

(b) F(aA+ bI) =aF(A) +5 foranya,b € Cand Ae C"". 

(c) F(A) C {a € C:a+4 > 0} ifand only if A + A* is positive semidefinite. 
Then F(A) = W(A) for all A € C"™". 


. Use the usual topology on C”*" and the Hausdorff metric on two compact sets A, B of C defined by 


d(A,B) = max; max min|a—b|, max minla-—)| 
acA beB beB acA 


The mapping At» W(A) is continuous. 

Suppose f(x+iy) = (ax+by+c)+i(dx+ey+ f) for some real numbers a, b,c,d,e, f. Define 
f(H+iG) = (aH+bG+clI)+i(dH+eG-+ f 1) for any two Hermitian matrices H,G € C"*". 
We have 


W(f(H +iG)) = f(W(A)) = {f(x + iy) ix +iy € W(A)}. 


Let D = {a €C: |a| < 1}. Suppose f : D > C is analytic in the interior of D and continuous on 
the boundary of D. 
(a) If f(D) C Dand f(0) = 0, then W( f(A)) C D whenever W(A) C D. 


(b) If f(D) C Cy = {a € C:a+4 > O}, then W(f(A)) C Cy \ {Cf (0) + f (0))/2} whenever 
W(A) CD. 


Suppose @ : C’*" > C”*" isaunital positive linear map. Then W(¢(A)) C W(A) forall A € C"*”. 
[Pel75] Let @ : C"*” — C"*" be linear. Then 


W(A) = W(@(A)) forall Aec™" 
if and only if there is a unitary U € C”*” such that @ has the form 
Xt U*XU or X U*X'U. 


[Li87] Let @ : C"*" > C"*" be linear. Then w(A) = w(@(A)) forall A € C”*" ifand only if there 
exist a unitary U € C’*" and a complex unit jz such that ¢ has the form 


Xpw wU*XU or Xb pU*X'U. 
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Much is known about spectral properties of square (complex) matrices. There are extensive studies of 
eigenvalues of matrices in certain classes. Some of the studies concentrate on the inertia of the matrices, 
that is, distribution of the eigenvalues in half-planes. 

A special inertia case is of stable matrices, that is, matrices whose spectrum lies in the open left or right 
half-plane. These, and other related types of matrix stability, play an important role in various applications. 
For this reason, matrix stability has been intensively investigated in the past two centuries. 

A. M. Lyapunov, called by E R. Gantmacher “the founder of the modern theory of stability,” studied 
the asymptotic stability of solutions of differential systems. In 1892, he proved a theorem that was restated 
(first, apparently, by Gantmacher in 1953) as a necessary and sufficient condition for stability of a matrix. 
In 1875, E. J. Routh introduced an algorithm that provides a criterion for stability. An independent solution 
was given by A. Hurwitz. This solution is known nowadays as the Routh—Hurwitz criterion for stability. 
Another criterion for stability, which has a computational advantage over the Routh—Hurwitz criterion, 
was proved in 1914 by Liénard and Chipart. The equivalent of the Routh—Hurwitz and Liénard—Chipart 
criteria was observed by M. Fujiwara. The related problem of requiring the eigenvalues to be within the 
unit circle was solved separately in the early 1900s by I. Schur and Cohn. The above-mentioned studies 
have motivated an intensive search for conditions for matrix stability. 

An interesting question, related to stability, is the following one: Given a square matrix A, can we find 
a diagonal matrix D such that the matrix DA is stable? This question can be asked in full generality, as 
suggested above, or with some restrictions on the matrix D, such as positivity of the diagonal elements. 
A related problem is characterizing matrices A such that for every positive diagonal matrix D, the matrix 
DA is stable. Such matrices are called multiplicative D-stable matrices. This type of matrix stability, as 
well as two other related types, namely additive D-stability and Lyapunov diagonal (semi)stability, have 
important applications in many disciplines. Thus, they are very important to characterize. While regular 
stability is a spectral property (it is always possible to check whether a given matrix is stable or not by 
evaluating its eigenvalues), none of the other three types of matrix stability can be characterized by the 
spectrum of the matrix. This problem has been solved for certain classes of matrices. For example, for 
Z-matrices all the stability types are equivalent. Another case in which these characterization problems 
have been solved is the case of acyclic matrices. 
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Several surveys handle the above-mentioned types of matrix stability, e.g., the books [HJ91] and [KB00], 
and the articles [Her92], [Her98], and [BH85]. Finally, the mathematical literature has studies of other types 
of matrix stability, e.g., the above-mentioned Schur—Cohn stability (where all the eigenvalues lie within 
the unit circle), e.g., [Sch17] and [Zah92]; H-stability, e.g., [OS62], [Car68], and [HM98]; L-stability 
and strict H-stability, e.g., [Tad81]; and scalar stability, e.g., [HM98]. 


19.1 Inertia 


Much is known about spectral properties of square matrices. In this chapter, we concentrate on the 
distribution of the eigenvalues in half-planes. In particular, we refer to results that involve the expression 
AH + HA*, where A is a square complex matrix and H is a Hermitian matrix. 


Definitions: 


For a square complex matrix A, we denote by (A) the number of eigenvalues of A with positive real part, 
by 6(A) the number of eigenvalues of A on the imaginary axis, and by v(A) the number of eigenvalues of 
A with negative real part. The inertia of A is defined as the triple in(A) = (2 (A), v(A), 6(A)). 


Facts: 
All the facts are proven in [OS62]. 


1. Let A be a complex square matrix. There exists a Hermitian matrix H such that the matrix AH + 
H A* is positive definite if and only if 5(A) = 0. Furthermore, in such a case the inertias of A and 
H are the same. 

2. Let {Aj,... An} be the eigenvalues of an n x n matrix A. If Wij Qi + Aj) # 0, then for any 
positive definite matrix P there exists a unique Hermitian matrix H such that AH + HA* = P. 
Furthermore, the inertias of A and H are the same. 

3. Let A be a complex square matrix. We have 6(A) = (A) = 0 if and only if there exists ann x n 


positive definite Hermitian matrix such that the matrix —(AH + H A*) is positive definite. 


Examples: 


1. It follows from Fact 1 above that a complex square matrix A has all of its eigenvalues in the right 
half-plane if and only if there exists a positive definite matrix H such that the matrix AH + H A* is 
positive definite. This fact, associating us with the discussion of the next section, is due to Lyapunov, 
originally proven in [L1892] for systems of differential equations. The matrix formulation is due 
to [Gan60]. 

2. In order to demonstrate that both the existence and uniqueness claims of Fact 2 may be false without 
the condition on the eigenvalues, consider the matrix 


1 0 
0 -1)’ 


for which the condition of Fact 2 is not satisfied. One can check that the only positive definite 
matrices P for which the equation AH + H A* = P has Hermitian solutions are matrices of the 


2 
pate > Pi» P22 > 0. Furthermore, for P = | 


A= 


0 pr 0 
Hermitian solutions of AH + H A* = P are all matrices H of the type 


type P = | ‘ it is easy to verify that the 
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If we now choose 


c 


then here the condition of Fact 2 is satisfied. Indeed, for H = : b 


2a —c 
—t —4b|’ 


which can clearly be solved uniquely for any Hermitian matrix P; specifically, for P = 


| we have 


AH+HA*= 


2 0 
0 4’ 


: ie : _|l 0 
the unique Hermitian solution H of AH + HA* = P is | — , ‘ 


19.2 Stability 


Definitions: 
A complex polynomial is negative stable [positive stable] if its roots lie in the open left [right] half-plane. 
A complex square matrix A is negative stable [positive stable] if its characteristic polynomial is negative 
stable [positive stable]. 

We shall use the term stable matrix for positive stable matrix. 

For an n X n matrix A and for an integer k, 1 < k <n, we denote by S;(A) the sum of all principal 
minors of A of order k. 

The Routh—-Hurwitz matrix associated with A is defined to be the matrix 


S\(A) S3(A) Ss(A) —- : é : 0 0 
1 S2(A) S4(A) 
0 = S\(A)_ S3(A) 
0 1 Sy(A) 
0 0 Si (A) . 
5 : 0 0 
Sn(A) 0 
: ; ; Sn-1(A) 0 
0 0 0 . . * . Sn—2(A) Sn(A) 


A square complex matrix is a P-matrix if it has positive principal minors. 

A square complex matrix is a P*-matrix if it has nonnegative principal minors and at least one principal 
minor of each order is positive. 

A principal minor of a square matrix is a leading principal minor if it is based on consecutive rows and 
columns, starting with the first row and column of the matrix. 

Ann x n real matrix A is sign symmetric if it satisfies 


det Ala, B] det A[B,a] > 0, Va, B C {1,...,n}, |oa| = |B]. 
Ann x n real matrix A is weakly sign symmetric if it satisfies 
det Ala, B] det A[B,a]>0, Va, BC {l,... .n}, lol =|] = la Bl +1. 


A square real matrix is a Z-matrix if it has nonpositive off-diagonal elements. 
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A Z-matrix with positive principal minors is an M-matrix. (See Section 24.5 for more information and 
an equivalent definition.) 


Facts: 


Lyapunov studied the asymptotic stability of solutions of differential systems. In 1892 he proved in his 
paper [L1892] a theorem which yields a necessary and sufficient condition for stability ofa complex matrix. 
The matrix formulation of Lyapunov’s Theorem is apparently due to Gantmacher [Gan60], and is given 
as Fact 1 below. The theorem in [Gan60] was proven for real matrices; however, as was also remarked in 
[Gan60], the generalization to the complex case is immediate. 


L: 


10. 


The Lyapunov Stability Criterion: A complex square matrix A is stable if and only if there exists a 
positive definite Hermitian matrix H such that the matrix AH + H A* is positive definite. 


. [OS62] A complex square matrix A is stable if and only if for every positive definite matrix G there 


exists a positive definite matrix H such that the matrix AH + HA* = G. 


. [R1877], [H1895] The Routh—Hurwitz Stability Criterion: An 1 x n complex matrix A with a real 


characteristic polynomial is stable if and only if the leading principal minors of the Routh—Hurwitz 
matrix associated with A are all positive. 


. [LC14] (see also [Fuj26]) The Liénard—Chipart Stability Criterion: Let A be an n x n complex 


matrix with a real characteristic polynomial. The following are equivalent: 

(a) A is stable. 

(b) S,(A), S,-2(A),... > 0 and the odd order leading principal minors of the Routh—Hurwitz 
matrix associated with A are positive. 

(c) S,(A), S,-2(A),... > 0 and the even order leading principal minors of the Routh—Hurwitz 
matrix associated with A are positive. 

(d) S,(A), Sy—1(A), Sn_3(A),... > O and the odd order leading principal minors of the Routh— 
Hurwitz matrix associated with A are positive. 


(e) S,(A), Sp—1(A), Sp—3(A),... > 0 and the even order leading principal minors of the Routh— 
Hurwitz matrix associated with A are positive. 


. [Car74] Sign symmetric P-matrices are stable. 


HK2003] Sign symmetric stable matrices are P -matrices. 

Hol99] Weakly sign symmetric P -matrices of order less than 6 are stable. Nevertheless, in general, 
weakly sign symetric P-matrices need not be stable. 

(For example, [BVW78]) A Z-matrixis stable ifand only ifit isa P -matrix (thatis, itisan M-matrix). 
FHRO5] Let A bea stable real square matrix. Then either all the diagonal elements of A are positive 
or A has at least one positive diagonal element and one positive off-diagonal element. 

FHRO5] Let ¢ be an n-tuple of complex numbers, n > 1, consisting of real numbers and conjugate 
pairs. There exists a real stable n x n matrix A with exactly two positive entries such that ¢ is the 
spectrum of A. 


Examples: 


1. 


Let 
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It is immediate to check that the latter matrix has positive leading principal minors. It, thus, follows 
that A is stable. Indeed, the eigenvalues of A are 1.4515, 0.0657, and 10.4828. 
2. Stable matrices do not form a convex set, as is easily demonstrated by the stable matrices 


Gor lear 


Deh : 3 , 
whose sum eS has eigenvalues —1 and 5. Clearly, convex sets of stable matrices do exist. An 


example of such a set is the set of upper (or lower) triangular matrices with diagonal elements in the 
open right half-plane. Nevertheless, there is no obvious link between matrix stability and convexity 
or conic structure. Some interesting results on stable convex hulls can be found in [Bia85], [FB87], 
[FB88], [CL97], and [HS90]. See also the survey in [Her98]. 

3. In view of Facts 5 and 7 above, it would be natural to ask whether stability of a matrix implies that 
the matrix is a P -matrix or a weakly sign symmetric matrix. The answer to this question is negative 
as is demonstrated by the matrix 

-1 1 
—5 3] 0 


The eigenvalues of A are 1 +i, and so A is stable. Nevertheless, A is neither a P -matrix nor a weakly 
sign symmetric matrix. 

4, Sign symmetric Po" -matrices are not necessarily stable, as is demonstrated by the sign symmetric 
P,t-matrix 


A= 


> 

II 
= 
ocooro 
Hoooco 
coorco 
oroo°o 


2ni 


The matrix A is not stable, having the eigenvalues 4 e*'3 ,1,1,1>. 


5. A P-matrix is not necessarily stable as is demonstrated by the matrix 


OW Fe 
wre © 


3 
0 
1 


For extensive study of spectra of P-matrices look at [HB83], [Her83], [HJ86], [HS93], and 
[HK2003]. 


19.3. Multiplicative D-Stability 


Multiplicative D-stability appears in various econometric models, for example, in the study of stability of 
multiple markets [Met45]. 


Definitions: 


A real square matrix A is multiplicative D-stable if D A is stable for every positive diagonal matrix D. 
In the literature, multiplicative D-stable matrices are usually referred to as just D-stable matrices. 
A real square matrix A is inertia preserving if the inertia of AD is equal to the inertia of D for every 
nonsingular real diagonal matrix D. 
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The graph G(A) of an n x n matrix A is the simple graph whose vertex set is {1,... ,n}, and where 
there is an edge between two vertices i and j (i # j) if and only ifa;; 4 0 or aj; # 0. (See Chapter 28 
more information on graphs.) 

The matrix A is said to be acyclic if G(A) is a forest. 


Facts: 


The problem of characterizing multiplicative D-stabity for certain classes and for matrices of order less 
than 5 is dealt with in several publications (e.g., [Cai76], [CDJ82], [Cro78], and [Joh74b]). However, in 
general, this problem is still open. Multiplicative D-stability is characterized in [BH84] for acyclic matrices. 
That result generalizes the handling of tridiagonal matrices in [CDJ82]. Characterization of multiplicative 
D-stability using cones is given in [HSh88]. See also the survey in [Her98]. 


1. Tridiagonal matrices are acyclic, since their graphs are paths or unions of disjoint paths. 
2. [FF58] For a real square matrix A with positive leading principal minors there exists a positive 
diagonal matrix D such that DA is stable. 
3. [Her92] For a complex square matrix A with positive leading principal minors there exists a positive 
diagonal matrix D such that DA is stable. 
4. [Cro78] Multiplicative D-stable matrices are P,' -matrices. 
5. [Cro78] A 2 x 2 real matrix is multiplicative D-stable if and only if it is a Pj‘ -matrix. 
6. [Cai76] A 3 x 3 real matrix A is multiplicative D-stable if and only if A + D is multiplicative 
D-stable for every nonnegative diagonal matrix D. 
7. [Joh75] A real square matrix A is multiplicative D-stable if and only if A + 7D is nonsingular for 
every positive diagonal matrix D. 
8. (For example, [BVW78]) A Z-matrix is multiplicative D-stable if and only if it is a P-matrix (that 
is, it is an M-matrix). 
9. [BS91] Inertia preserving matrices are multiplicative D-stable. 
10. [BS91] An irreducible acyclic matrix is multiplicative D-stable if and only if it is inertia preserving. 
11. [HK2003] Let A be a sign symmetric square matrix. The following are equivalent: 


(a) The matrix A is stable. 

(b) The matrix A has positive leading principal minors. 
(c) The matrix A is a P-matrix. 

(d) The matrix A is multiplicative D-stable. 


(e) There exists a positive diagonal matrix D such that the matrix DA is stable. 


Examples: 


1. In order to illustrate Fact 2, let 


1 1 
A= |0 1]. 
4 2 


The matrix A is not stable, having the eigenvalues 4.0606 and —0.0303 + 0.49531. Nevertheless, 
since A has positive leading minors, by Fact 2 there exists a positive diagonal matrix D such that 
the matrix DA is stable. Indeed, the eigenvalues of 


1 0 O 1 1 1 1 1 1 
0 1 =O 0 1 1] =]0 1 1 
0 0 O01) |4 1 2 0.4 0.1 0.2 


are 1.7071, 0.2929, and 0.2. 
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2. In order to illustrate Fact 4, let 


The matrix A is stable, having the eigenvalues 0.3376 + 0.56231 and 2.3247. Yet, we have det 
A[{2, 3}] < 0, and so A is nota Pit -matrix. Indeed, observe that the matrix 


0.1 0 O 1 1 0 0.1 0.1 0 
Oo 1 0; |-1 0 1/=]-1 O 1 
0 O01 0 1 2 0 1 2 


is not stable, having the eigenvalues —0.1540 + 0.13357 and 2.408. 

3. While stability is a spectral property, and so it is always possible to check whether a given matrix 
is stable or not by evaluating its eigenvalues, multiplicative D-stability cannot be characterized by 
the spectrum of the matrix, as is demonstrated by the following two matrices 


1 0 —-1 2 
0 AF a= | i 


The matrices A and B have the same spectrum. Nevertheless, while A is multiplicative D-stable, 
B is not, since it is nota Pot -matrix. Indeed, the matrix 


eer eae 


has eigenvalues —0.5 + 3.12253. 
4. It is shown in [BS91] that the converse of Fact 9 is not true, using the following example from 


A= 


[Har80]: 
0  —50 
A= 1 0 
1 1 


The matrix A is multiplicative D-stable (by the characterization of 3 x 3 multiplicative D-stable 
matrices, proven in [Cai76]). However, for D = diag (—1,3,—1) the matrix AD is stable and, 
hence, A is not inertia preserving. In fact, it is shown in [BS91] that even P-matrices that are both 
D-stable and Lyapunov diagonally semistable (see section 19.5) are not necessarily inertia preserving. 


19.4 Additive D-Stability 


Applications of additive D-stability may be found in linearized biological systems, e.g., [Had76]. 


Definitions: 


A real square matrix A is said to be additive D-stable if A + D is stable for every nonnegative diagonal 
matrix D. 
In some references additive D-stable matrices are referred to as strongly stable matrices. 


Facts: 


The problem of characterizing additive D-stability for certain classes and for matrices of order less than 5 
is dealt with in several publications (e.g., [Cai76], [CDJ82], [Cro78], and [Joh74b]). However, in general, 
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this problem is still open. Additive D-stability is characterized in [Her86] for acyclic matrices. That result 
generalizes the handling of tridiagonal matrices in [Car84]. 


1. [Cro78] Additive D-stable matrices are Po -matrices. 

2. [Cro78] A 2 x 2 real matrix is additive D-stable if and only if it is a Pp‘ -matrix. 

3. [Cro78] A 3 x 3 real matrix A is additive D-stable if and only if it is a Po*-matrix and stable. 

4, (For example, [BVW78]) A Z-matrix is additive D-stable if and only if it is a P-matrix (that is, it 
is an M-matrix). 

. An additive D-stable matrix need not be multiplicative D-stable (cf. Example 3). 

[Tog80] A multiplicative D-stable matrix need not be additive D-stable. 


nun 


Examples: 


1. In order to illustrate Fact 1, let 


The matrix A is stable, having the eigenvalues 0.3376 + 0.56237 and 2.3247. Yet, we have det A[2, 3|2, 3] 
< 0, and so A is nota Pot -matrix. Indeed, observe that the matrix 


0 3 1 
0} =|-1 0 
0 0 ol 


a) 


is not stable, having the eigenvalues 2.5739 + 0.36907 and —0.1479. 

2. While stability is a spectral property, and so it is always possible to check whether a given matrix 
is stable or not by evaluating its eigenvalues, additive D-stability cannot be characterized by the 
spectrum of the matrix, as is demonstrated by the following two matrices: 


1 0 —-1 2 
0 Ae a= |) i 


The matrices A and B have the same spectrum. Nevertheless, while A is additive D-stable, B is 
not, since it is nota Py -matrix. Indeed, the matrix 


a +e l-[3 3 


has eigenvalues —0.1623 and 6.1623. 
3. In order to demonstrate Fact 5, consider the matrix 


A= 


0.25 1 O 
A= |-1 05 Il], 
2.1 1 2 


which is a Po’ matrix and is stable, having the eigenvalues 0.0205709 + 1.23009i and 2.70886. 
Thus, A is additively D-stable by Fact 3. Nevertheless, A is not multiplicative D-stable, as the 
eigenvalues of 


1 0 O; {0.25 1 O 0.25 1 O 
0 5 0 —-1 05 1)/=]-5 2.5 5 
0 O 4] | 2.1 1 2 84 4 8 


are —0.000126834 + 2.761837 and 10.7503. 
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19.5 Lyapunov Diagonal Stability 


Lyapunov diagonally stable matrices play an important role in various applications, for example, predator— 
prey systems in ecology, e.g., [Goh76], [Goh77], and [RZ82]; dynamical systems, e.g., [Ara75]; and eco- 
nomic models, e.g., [Joh74a] and the references in [BBP78]. 


Definitions: 


A real square matrix A is said to be Lyapunov diagonally stable [semistable] if there exists a positive 
diagonal matrix D such that AD + DA? is positive definite [semidefinite]. In this case, the matrix D is 
called a Lyapunov scaling factor of A. 

In some references Lyapunov diagonally stable matrices are referred to as just diagonally stable matrices 
or as Volterra—Lyapunov stable. 

Ann Xx n matrix A is said to be an H-matrix if the comparison matrix M(A) defined by 


Slab t= 5 
may = {Hi ix; 


is an M-matrix. 
A real square matrix A is said to be strongly inertia preserving if the inertia of AD is equal to the 
inertia of D for every (not necessarily nonsingular) real diagonal matrix D. 


Facts: 


The problem of characterizing Lyapunov diagonal stability is, in general, an open problem. It is solved in 
[BH83] for acyclic matrices. Lyapunov diagonal semistability of acyclic matrices is characterized in [Her88]. 
Characterization of Lyapunov diagonal stability and semistability using cones is given in [HSh88]; see also 
the survey in [Her98]. For a book combining theoretical results, applications, and examples, look at [KBO00]. 


1. [BBP78], [Ple77] Lyapunov diagonally stable matrices are P-matrices. 

2. [Goh76] A 2 x 2 real matrix is Lyapunov diagonally stable if and only if it is a P-matrix. 

3. [BVW78] A real square matrix A is Lyapunov diagonally stable if and only if for every nonzero 
real symmetric positive semidefinite matrix H, the matrix H A has at least one positive diagonal 
element. 

4. [QR65] Lyapunov diagonally stable matrices are multiplicative D-stable. 

5. [Cro78] Lyapunov diagonally stable matrices are additive D-stable. 

6. [AK72], [Tar71] A Z-matrix is Lyapunov diagonally stable if and only if it is a P-matrix (that is, it 
is an M-matrix). 

7. [HS85a] An H-matrix A is Lyapunov diagonally stable if and only if A is nonsingular and the 
diagonal elements of A are nonnegative. 

8. [BS91] Lyapunov diagonally stable matrices are strongly inertia preserving. 

9. [BH83] Acyclic matrices are Lyapunov diagonally stable if and only if they are P-matrices. 

10. [BS91] Acyclic matrices are Lyapunov diagonally stable if and only if they are strongly inertia 
preserving. 


Examples: 


1. Multiplicative D-stable and additive D-stable matrices are not necessarily diagonally stable, as is 
demonstrated by the matrix 
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2. Another example, given in [BH85] is the matrix 


0 1 0 0 
at 0 

> 
Oe Bt Ree, 
0 0 -b 0 


which is not Lyapunov diagonally stable, but is multiplicative D-stable if and only if a > 1, and is 
additive D-stable whenever a = 1 andb 1. 

3. Stability is a spectral property, and so it is always possible to check whether a given matrix is stable 
or not by evaluating its eigenvalues; Lyapunov diagonal stability cannot be characterized by the 
spectrum of the matrix, as is demonstrated by the following two matrices: 


1 0 ait 2 
0 i a= [3 al 


The matrices A and B have the same spectrum. Nevertheless, while A is Lyapunov diagonal stable, 

B is not, since it is not a P-matrix. Indeed, for every positive diagonal matrix D, the element of 

AD + DA? in the (1, 1) position is negative and, hence, AD + DA! cannot be positive definite. 
4. Let A be a Lyapunov diagonally stable matrix and let D be a Lyapunov scaling factor of A. Using 


A= 


continuity arguments, it follows that every positive diagonal matrix that is close enough to D is 
a Lyapunov scaling factor of A. Hence, a Lyapunov scaling factor of a Lyapunov diagonally stable 
matrix is not unique (up to a positive scalar multiplication). The Lyapunov scaling factor is not 
necessarily unique even in cases of Lyapunov diagonally semistable matrices, as is demonstrated by 
the zero matrix and the following more interesting example. Let 


PN N 
NO WwW Ww 


One can check that D = diag (1, 1,d) is a scaling factor of A whenever ; < d < 1. On the other 
hand, it is shown in [HS85b] that the identity matrix is the unique Lyapunov scaling factor of the 


matrix 
1 1 2 0 
1 1 0 0 
0 2 1 2I- 
2 2 0 1 


Further study of Lyapunov scaling factors can be found in [HS85b], [HS85c], [SB87], [HS88], 
[SH88], [SB88], and [CHS92]. 
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In general, an inverse eigenvalue problem (IEP) consists of the construction of a matrix with prescribed 
structural and spectral constraints. This is a two-level problem: (1) on a theoretical level the target is 
to determine if the IEP is solvable, that is, to find necessary and sufficient conditions for the existence 
of at least one solution matrix (a matrix with the given constraints); and (2) on a practical level, the 
target is the effective construction of a solution matrix when the IEP is solvable. IEPs are classified into 
different types according to the specific constraints. We will consider three topics: IEPs with prescribed 
entries, nonnegative IEPs, and affine parameterized IEPs. Other important topics include pole assignment 
problems, Jacobi IEPs, inverse singular value problems, etc. For interested readers, we refer to the survey 
[CG02] where an account of IEPs with applications and extensive bibliography can be found. 


20.1 IEPs with Prescribed Entries 


The underlying question for an IEP with prescribed entries (PEIEPs) is to understand how the prescription 
ofsome entries ofa matrix can have repercussions on its spectral properties. A classical result on this subject 
is the Schur—Horn Theorem allowing the construction ofa real symmetric matrix with prescribed diagonal, 
prescribed eigenvalues, and subject to some restrictions (see Fact 1 below). Here we consider PEIEPs that 
require finding a matrix with some prescribed entries and with prescribed eigenvalues or characteristic 
polynomial; no structural constraints are imposed on the solution matrices. 

Most of the facts of Sections 20.1 and 20.2 appear in [IC00], an excellent survey that describes finite 
step procedures for constructing solution matrices. 
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Definitions: 


An IEP with prescribed entries (PEIEP) has the following standard formulation: 
Given: 


(a) A field F. 

(b) nelements Aj,...,4, of F (respectively, a monic polynomial f € F [x] of degree n). 
(c) telements pj,..., p; of F. 

(d) Aset O = {(i), j1),..-. 5 (its je)} of t positions of ann x n matrix. 


Find: A matrix A = [aj;j] € F"*" with aj,;, = px for 1 < k < t and such that o(A) = {A1,... An} 
(respectively, such that p4(x) = f). 


Facts: [IC00] 


1. (Schur—Horn Theorem) Given any real numbers A; > --- > 4, and d; > --- > d,, satisfying 


k k 
yg = Ya fork =1,...,n—1 and Ss 
i=l i=l i=l i=l 


there exists a real symmetric n x n matrix with diagonal (d),... ,d,) and eigenvalues A,... An; 
and any Hermitian matrix satisfies these conditions on its eigenvalues and diagonal entries. 

2. A finite step algorithm is provided in [CL83] for the construction of a solution matrix for the 
Schur—Horn Theorem. 

3. Consider the following classes of PEIEPs: 


(1.1) F Misses a An Pis-++ > Pn-1 |Ol=n—-1 
(1.2) F fax teyx™ 4. ten Plo-++ > Pn-1 |QjJ=n—-1 
(2.1) F Ais An Plo--+>Pn |Ql=n 
(2.2) F f =a" eyxt! ay Cn Pio- oe >Pn |Q| =n 
(3.1) F Alye+2 An Plo-++ > P2n—3 |Q| = 2n —3 


¢ {dO73a] Each PEIEP of class (1.1) is solvable. 


¢ [Dds74] Each PEIEP of class (1.2) is solvable except if all off-diagonal entries in one row or 
column are prescribed to be zero and f has no root on F. 
* [dO73b] Each PEIEP of class (2.1) is solvable with the following exceptions: (1) all entries in 
the diagonal are prescribed and their sum is different from A, + --- + A,; (2) all entries in one 
row or column are prescribed, with zero off-diagonal entries and diagonal entry different from 
Aq. +. An; and (3) n = 2, O = {(1,2), (2, 1)}, and x? — (Ay +A2)x + pips +Aya2 € F[x] is 
irreducible over F. 
[Zab86] For n > 4, each PEIEP of class (2.2) is solvable with the following exceptions: (1) all 
entries in the diagonal are prescribed and their sum is different from-c,; (2) all entries in a row 
or column are prescribed, with zero off-diagonal entries and diagonal entry which is not a root 
of f; and (3) all off-diagonal entries in one row or column are prescribed to be zero and f has 
no root on F. The case n < 4 is solved but there are more exceptions. 
[Her83] Each PEIEP of class (3.1) is solvable with the following exceptions: (1) all entries in the 
diagonal are prescribed and their sum is different from A, + --- + 4,3 and (2) all entries in one 
row or column are prescribed, with zero off-diagonal entries and diagonal entry different from 
Misexteh Aas 
[Her83] The result for PEIEPs of class (3.1) cannot be improved to |Q| > 2n — 3 since a lot of 
specific nonsolvable situations appear, and, therefore, a closed result seems to be quite inaccessible. 


A gradient flow approach is proposed in [CDS04] to explore the existence of solution matrices 
when the set of prescribed entries has arbitrary cardinality. 
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4, The important case Q = {(i, j) :i # j} is discussed in section 20.9. 

5. Let {pjj :1 <i < j < n} bea set of win elements of a field F. Define the set {r),... ,7;} of all 
those integers r such that pj; = 0 whenever 1 <i <r < j < n.Assumethat0 = 19 <r; <--- < 
rs <s41 = nand define B,; = Der <k <r, Dkk fort = 1,...,5 +1. The following PEIEPs have 
been solved: 


* [BGRS90] Let A;,... ,A, be m elements of F. Then there exists A = [a;;] € F"*" with ajj = pij 
forl <i < j < mand o(A) = {Aj,...,A,} if and only if {1,...,m} has a partition 
N, U-++U N,4; such that |N,| =r; — 17-1 and > hen, Ax = B, for eacht =1,...,5 41. 

* [Sil93] Let f € F[x] be a monic polynomial of degree n. Then there exists A = [ajj] € F"*” 
with ajj = pij for 1 <i < j < mand pa(x) = f ifand only if f = f--- fe41, where 
freaxtete — Bx til... € F[x]fort=1,...,5+1. 


6. [Fil69] Let d),...,d, be elements of a field F, and let A € F"*" with A # AI, for all A € F and 
tr(A) = 7y_, d;. Then A is similar to a matrix with diagonal (d),... , dn). 


Examples: 
1. [dO73b] Given: 
(a) A field F. 


(b) Ay,...5,A, € F. 
(c) Pi,---> Pn € F. 


(d) Q={(1,1),...,(a,n)}. 
iar Ai = Deny Pi> then A = [aij] € F"™" with 


Gii = Pir aj; =0 if i<j-2, 


i i 
aii = > de —>— Pee aij =pj-Aju if i> jf, 
k=1 k=1 


has diagonal (p1,... , py) and its spectrum is 0(A) = {Aj,... 5 An}. 


20.2 PEIEPs of 2 x 2 Block Type 


In the 1970s, de Oliveira posed the problem of determining all possible spectra of a 2 x 2 block matrix 
A or all possible characteristic polynomials of A or all possible invariant polynomials of A when some 
of the blocks are prescribed and the rest vary (invariant polynomial is a synonym for invariant factor, 
cf. Section 6.6). 


Definitions: 
Let F bea field and let A be the 2 x 2 block matrix 


Au Az 


A= 
Ad Ax 


eF™" with Ay, ¢F*! and Ane F™™", 


Notation: 
* deg( f): degree of f € F[x]. 
* g\|f: polynomial g divides the polynomial f. 
* ip(B): invariant polynomials of the square matrix B. 
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IC00] 


dO71] Let Aj; anda monic polynomial f € F [x] of degree n be given. Let ip(Ai1) = gil---|gz. 
Then pa(x) = f is possible except if] > mand gy --- gj—-m is not a divisor of f. 

dS79], [Tho79] Let A,,; and m monic polynomials fi,..., f, € F[x] with fi|---|f, and 
Sj deg( fi) = n be given. Let ip(Ai) = gil---|gi. Then ip(A) = fil---| fy is possible if 
and only if fj | gi | fi+2m for eachi = 1,...,] where fy = 0 fork > n. 

dO75] Let Ajz and a monic polynomial f € F[x] of degree n be given. Then pa(x) = f is 
possible except if Ajz = 0 and f has no divisor of degree I. 

Zab89], [Sil90] Let Ay, andn monic polynomials fi,... , f, € F[x] with f\|---|f,and >7j_, deg 
(fi) = n be given. Let r = rank(A))) and s the number of polynomials in fi,... , f, which are 
different from 1. Then ip(A) = fi|---| fy is possible if and only if r < n — s with the following 
exceptions: 


(a) r = Oand[]j_, fj has no divisor of degree I. 
(b) r > 1,1 —r oddand f,_,4; =--- = f, with f, irreducible of degree 2. 
(c) r = land f,-;41 =--: = f, with f, irreducible of degree k > 3 and k{I. 


. [Wim74] Let Aj, Ay, and a monic polynomial f € F [x] of degree n be given. Let h,|---|h; be 


the invariant factors of [xl —Ay| —Ap | . Then pa(x) = f is possible ifand only if hy -- - hj| f. 


. All possible invariant polynomials of A are characterized in [Zab87] when Aj; and Aj) are given. 


The statement of this result contains a majorization inequality involving the controllability indices 
of the pair (Aj;, Aj). 


. [Sil87b] Let Ay), Ay2, and n elements Aj,...,A, of F be given. Assume that ] > m and let 


ip(An) = gil--- |g. Then o(A) = {A1,... , An} is possible if and only if all the following condi- 
tions are satisfied: 


(a) tr(Ay,) + tr(A22) = Ay +++ F Ay. 
(b) If] > m, then gy -+ + gj-m|(x — Aq) - ++ (x — Ay). 


(c) If Ay) = al) and Ay = dI,,, then there exists a permutation t of {1,...,m} such that 
Ar(2i-1) + Ari) =a +dforl<i< mand i1(;) =a for2m+1< j <n. 


. [Sil87a] Let Aj, Ag}, and n elements Aj,...,A, of F be given. Then o(A) = {Aq,... An} is 


possible except if, simultaneously, /] = m = 1, Ajo = [b], Ax = [c] and the polynomial 
x? — (Ay + Ax)x + be + AyA2 € F [x] is irreducible over F. 


. Let Ajo, Ag}, and a monic polynomial f € F [x] of degree n be given: 


¢ [Fri77] If F is algebraically closed then p4(x) = f is always possible. 

* [MSO00] If F = Rand n > 3 then pa(x) = f is possible if and only if either min{rank(Aj2), 
rank(A2;)} > 0 or f has a divisor of degree /. 

* If F =R, Ap =[b], An =[c] and f = x? +¢1x + cz € R[x] then p,(x) = f is possible 
if and only if x? + c)x +c. + be hasa root inR. 


. [Sil91] Let Ajj, Aj, Ag2, and n elements Aj,...,A, of F be given. Let k,|---|ky be the in- 


variant factors of [xl — Ay | —- An], h,|---|h» the invariant factors of areal and 

g = ky---kjhy--+hy. Then o(A) = {Aj,...,An} is possible if and only if all the following 

conditions hold: 

(a) tr(Ay,) + tr(Ag2) = Ay + +++ + Ag. 

(b) g(x — Ai) +++ (% — An). 

(c) If Ai: Aiz2 + AizA22 = n Ai for some n € F, then there exists a permutation t of {1,... , 1} 
such that A,@j—1) + Arai) = n for 1 <i < t where tf = rank(Ajp) and A7(2741),... 5 Ar(n) are 
the roots of g. 


. Ifa problem of block type is solved for prescribed characteristic polynomial then the solution for 


prescribed spectrum easily follows. 
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12. The book [GKvS95] deals with PEIEPs of block type from an operator point of view. 
13. A description is given in [FS98] of all the possible characteristic polynomials of a square matrix 
with an arbitrary prescribed submatrix. 


20.3  Nonnegative IEP (NIEP) 


Nonnegative matrices appear naturally in many different mathematical areas, both pure and applied, such 
as numerical analysis, statistics, economics, social sciences, etc. One of the most intriguing problems in this 
field is the so-called nonnegative IEP (NIEP). Its origin goes back to A.N. Kolgomorov, who in 1938 posed 
the problem of determining which individual complex numbers belong to the spectrum of some n x n 
nonnegative matrix with its spectral radius normalized to be 1. Kolgomorov’s problem was generalized in 
1949 by H. R. Suleimanova, who posed the NIEP: To determine which n-tuples of complex numbers are 
spectra of n x n nonnegative matrices. For definitions and additional facts about nonnegative matrices, 
see Chapter 9. 


Definitions: 


Let II, denote the compact subset of C bounded by the regular n-sided polygon inscribed in the unit circle 
of C and with one vertex at 1 € C. 

Let ©, denote the subset of C composed of those complex numbers 4 such that A is an eigenvalue of 
some n X n row stochastic matrix. 

A circulant matrix is a matrix in which every row is obtained by a single cyclic shift of the previous row. 


Facts: 
All the following facts appear in [Min88]. 


1. Acomplex nonzero number A is an eigenvalue of a nonnegative n x n matrix with positive spectral 
radius p if and only ifA/p € Oy. 

2. [DD45], [DD46] ©; = 1, UTIs. 

3. [Mir63] Each point in IT, U II; U--- UTI, is an eigenvalue of a doubly stochastic n x nm matrix. 

4, [Kar51] The set ©,, is symmetric relative to the real axis and is contained within the circle |z| < 1. 
It intersects |z| = 1 at the points e *s" where a and b run over all integers satisfying0 <a <b <n. 
The boundary of ©, consists of the curvilinear arcs connecting these points in circular order. For 
n > 4, each arc is given by one of the following parametric equations: 


zi(zP —t)' =(1—-1t)’, 
apes (aye 


where the real parameter tf runs over the interval 0 < ¢t < 1, andc,d, p,q,r are natural numbers 
defined by certain rules (explicitly stated in [Min88]). 


Examples: 
1. [LL78] The circulant matrix 


; 1+ 2r cosé 1—2rcos(} +6) 1-—2rcos(¥ — 6) 
- 1 — 2r cos(Z — @) 1+ 2rcos@ 1 — 2r cos(Z + @) 
1—2rcos(¥+@) 1 —2rcos(¥ — 6) 1+ 2r cos@ 


has spectrum {1, re’, re~‘*}, and it is doubly stochastic if and only if re’? € THs. 
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20.4 Spectra of Nonnegative Matrices 


Definitions: 
Ni, = {o = {Ay,... An} CC:45A> 0 with spectrum o}. 
Rn = {o = {dj,... An} CR:4A>0 _ with spectrum o}. 
S, = {o = {Aj,...,An} C R:3A>0 symmetric with spectrum o}. 


Re = {(1,Ao,... »An) eR": {1,Ag,... yAn} ER AW1 > Ang >i > An}. 
S; = {(1,Ao,... shy.) eR": {l,A2,... »An} € S,3 1 > Ang > > An}. 
For any set o = {Ay,... ,An} C C, let 


p(o) = max |A;| and 5 = sou for eachk EN. 
l<i<n fl 
A set S C R" is star-shaped from p € S if every line segment drawn from p to another point in S lies 
entirely in S. 


Facts: 
Most of the following facts appear in [ELN04]. 
1. [Joh81] Ifo = {A1,... ,An} € MN, then o is the spectrum of an x n nonnegative matrix with all 
row sums equal to p(c). 
2. Ifo = {Ay,... An} € N,,, then the following conditions hold: 
(a) plo) €o. 
(b) f=0. 
(c) s; > Ofori > 1. 
(d) [LL78], JJoh81] s”" < n'™~!s,,, fork,m > 1. 
3. N,, is known for n < 3, R,, and S,, are known for n < 4: 
* Ny = Ro = 82 = {0 = (A1,A2} CR: 5) > 0}. 
* R3 = 83 = {o = {Aj,A2,A3} CR: p(o) € 0; 8; = O}. 


* [LL78] N3 = {o = {A1,A2,A3} CC: 5 =9; p(o) € 03 8; > 0; st < 3s}. 


© Rg = Sy = {o = {A,A2,A3,A4} CR: p(o) € 0; 8; > O}. 
4. (a) [JLL96] FR, and S,, are not always equal sets. 
(b) [ELN04] o = {97,71, —44, —54, -70} € R; buto ¢ Ss. 
(c) [ELN04] provides symmetric matrices for all known elements of Ss. 
5, [Rea96] Let o = {A1,A2,A3,A4} C C with s, = 0. Theno € N, if and only if s; > 0, s3 > 0 and 
A4s4 > s3. Moreover, o is the spectrum of 
0 1 0 0 
& 0 1 0 


4s4—s5 ‘$83 Sy 0 
16 12 4 


6. [LM99] Let o = {Aj,A2,A3,A4,A5} C C with s; = 0. Theno e€ WN; if and only if the following 
conditions are satisfied: 


(a) s; > 0 fori = 2,3,4,5. 
(b) 4s4 > s5. 
(c) 1285 — 58283 + 583/484 — s3 > 0. 
The proof of the sufficient part is constructive. 
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7. (a) R* and S* are star-shaped from (1,... , 1). 
(b) [BM97] R; is star-shaped from (1,0,... ,0). 
(c) [KM01], [Mou03] R* and S* are not convex sets for n > 5. 


Examples: 


1. We show that o = {5,5, —3, —3, —3} ¢ Ns. Suppose A is a nonnegative matrix with spectrum o. 
By the Perron—Frobenius Theorem, A is reducible and o can be partitioned into two nonempty 
subsets, each one being the spectrum of a nonnegative matrix with Perron root equal to 5. This is 
not possible since one of the subsets must contain numbers with negative sum. 

2. {6, 1, 1,—4, —4} € Ns by Fact 6. 


20.5 Nonzero Spectra of Nonnegative Matrices 


For the definitions and additional facts about primitive matrices see Section 29.6 and Chapter 9. 


Definitions: 
The Mébius function  : N + {-1,0, 1} is defined by w(1) = 1, w(m) = (—1)° if m is a product of 
e distinct primes, and .(m) = 0 otherwise. 

The k'" net trace of o = {A,,... ,An} C Cistr(o) = Yak w(4)sa. 


The set o = {Aj,... ,An} C C with 0 ¢ o is the nonzero spectrum of a matrix if there exists a t x f 
matrix, t > n, whose spectrum is {Aj,... 5An,0,... ,0} with t — 1 zeros. 

The set o = {Ay,... ,An} C C has a Perron value if o(0) € o and there exists a unique index i with 
hi = plo). 
Facts: 


1. [BH91] Spectral Conjecture: Let S be a unital subring of R. The set o = {Aj,...,An} C C 
with 0 ¢ o is the nonzero spectrum of some primitive matrix over S if and only if the following 
conditions hold: 


(a) o has a Perron value. 

(b) All the coefficients of the polynomial [Jj_, (x — A;) lie in S. 

(c) IfS = Z, then tr,(o) > 0 for all positive integers k. 

(d) IfS # Z, then x > 0 for allk € Nands,, > 0 implies s,,) > 0 for all m, p € N. 


2. [BH91] Subtuple Theorem: Let S be a unital subring of R. Suppose that 0 = {Ay,... ,An} CC 
with 0 ¢ a has p(a) = A, and satisfies conditions (a) to (d) of the spectral conjecture. If for some 
j < nthe set {A,,... ,A;} is the nonzero spectrum of a nonnegative matrix over S, then o is the 
nonzero spectrum of a primitive matrix over S. 

3. The spectral conjecture is true for S = R by the subtuple theorem. 

4. [KOROO] The spectral conjecture is true forS = Z andS = Q. 

5. [BH91] The set o = {A1,... An} C C with 0 ¢ a is the nonzero spectrum of a positive matrix if 
and only if the following conditions hold: 


(a) o has a Perron value. 
(b) All coefficients of []j_,(x — 4;) are real. 
(c) sp > Oforallk EN. 
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Examples: 
1. Leto, = {5,4 + €, —3, —3, —3}. Then: 
(a) o. for € < 0 is not the nonzero spectrum of a nonnegative matrix since s, < 0. 
(b) oo is the nonzero spectrum of a nonnegative matrix by Fact 2. 
(c) o; is not the nonzero spectrum of a nonnegative matrix by arguing as in Example 1 of 
Section 20.4. 
(d) o, fore > 0,€ #1, is the nonzero spectrum of a positive matrix by Fact 5. 


20.6 Some Merging Results for Spectra 
of Nonnegative Matrices 


Facts: 

1. Tf{Aq,... An} © My and {111,... 5m} € Nn then {A1,... Ag) Mis--- Lem} € Naam: 

2. [Fie74] Leto = {A,...,An} € S, with p(o) = A, andt = {py,... Um} € Sm with p(t) = py. 
Then {A; + €,Ag,... 5 Ans M1 — € flas--- bem} © Sntm for any € > 0 if Ay > py. The proof is 
constructive. 

3. [Smi04] Let A bea nonnegative matrix with spectrum {A,,... ,A,,} and maximal diagonal element 
d, and let t = {j11,... 5m} € Nw with p(t) = wy. Ifd > py, then {Aq,... , Any M2... 5m} € 
Ni+m—1- The proof is constructive. 

4. Leto = {Aj,...,An} € N, with p(o) = A, and let € > 0. Then: 

(a) [Wuw97] {A; + 6€,A2,... An} € Mp. 
(b) IfA2 € R, then not always {A1,A2 + €,A3,--. An} € Nj, (see the previous example). 
(c) [Wuw97] If A2 € R, then {A; + €,A2 £6,A3,... ,An} © Mn (the proof is not constructive). 


Examples: 


1. Leto = {Aj,...,An} € NM, with p(o) = Ay, and tT = {ty,..., fm} € Nn with p(t) = pm. 
By Fact 1 of section 20.4 there exists A > 0 with spectrum o and row sums A,, and B > 0 with 
spectrum Tt and row sums /1;. [BMS04] If A, > jz; and e > 0, then the nonnegative matrix 


A |e ee? ( 
(A: — pi te)ee!] B = 
has row sums A; + € and spectrum {A; + €,A2,... Any M1 — © May. ++ > em}. 


20.7 Sufficient Conditions for Spectra 
of Nonnegative Matrices 


Definitions: 


The set {Aj,... ,Aj—1,, B, Ai4i,..- 5 An} is a negative subdivision of {A),...,A,} ifa + B = A; with 
a, BA; <0. 


Facts: 
Most of the following facts appear in [ELN04] and [SBM05]. 


1. [Sul49] Leto = {Ay,...,A,} C Rwitha, >--->A,.Theno € R, if 


eA SOS AQ SS An 
(Su) 
ed tes +An > 0 
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2: 


[BMS04] Complex version of (Su). Let o = {Aj,... An} C C bea set that satisfies: 
(a) T=0. 

(b) p(o) =A1. 

(c) Ar tes-+An = 0. 

(d) {Ag,... ,An} C {z € C: Rez < 0,|Rez| > |Imz|}. 


Then o € N,, and the proof is constructive. 


. [Sou83] Leto = {A1,... An} C R with A, > --- > A,. Then there exists a symmetric doubly 


stochastic matrix D such that A; D has spectrum o if 


1 n— An— —2k+2 n— 
(Sou) {a eS ray a+ G kane =° where m= [54] 


and the proof is constructive. 


. [Kel71] Leto = {Ay,... ,An} C Rwitha, > --- > d,. Letr be the greatest index for which A, > 0 


andlet6; = An42-; for2 < i < n—r+1.DefineK = {i : 2 <i < min{r,n—r+1} and A;+6; < 0}. 
Theno € R, if 


(Ke) @ Ai t+ Viek, jee Ai Oy) Gk = Oforalhh eK 
e 
© Ait View Ai + 81) + WHT) 4; = 0 


. [Bor95] Leto = {A),...,An} C Rwitha, >--->A,.Theno € R, if 


(Bo { dt = {f,,..., Ba} C R with d < n that satisfies conditions (Ke) 


e cis obtained from t after n — d negative subdivisions 


. [Sot03] Let o = {A,,...,4,} C R. For any partition o = o U--- Uo where o® = 


{A19,..- M9} with AY!) > --- > A{) define 


R\ ee +P ji for2<j< [=] and Roe Sa if n, odd; 
2 ys 
. (k) 
Fy 7 FAR + Lao R; : 


Theno € R, if 


there exists a partition o = 0) U--- Uo such that 
t 
Sot 1 k 1 k 
(Sot) eau » T' > max {aj — T, max{a{y} 
TW <0,k=2 ines 


and the proof is constructive. 


. [SBM05] (Su) = (Ke) => (Bo) = (Sot) and no opposite implication is true. 
. [Rad96] (Sou) and (Ke) are not comparable (see Example 2 below). 

. [Rad96] Ifo satisfies (Bo), theno € S,. 

10. 


[RS03] Leto = {Ay,... ,An} C C with o = o andd = — >}, A; > 0. Let cy,... ,c, be defined 
by (x — d) []fiy(x — Aj) = x" + hy eax” * If, = d(1+ aso Gb)» then o € N,,. The proof 
is constructive. 


Examples: 


1. 


Ifo = {Aj,... 5 An} C R satisfies (Su), then the companion matrix of the polynomial | ]j_, (x — A;) 
is nonnegative with spectrum o. 
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2. {5,3,—2, —2, —4} satisfies (Ke), but not (Sou), and {5, 3, —2, —2, —2, —2} satisfies (Sou), but not 
(Ke). 

3. o = {8,4, —3, —3, —3, —3} does not satisfies (Ke), but it satisfies (Bo) since o is obtained from 
T = {8,4, —6, —6} after two negative subdivisions and T satisfies (Ke). 

4. o = {9,7,2,2, —5, —5, —5, —5} does not satisfy (Bo), but it satisfies (Sot) witha!) = {9,2,—5,—5} 
and a2) = {7,2, —5, —5}. 

5. o = {6,1,1,—4, —4} does not satisfy (Sot), but o € Rs (Example 2 of section 20.4). 


20.8 Affine Parameterized IEPs (PIEPs) 


The set F"*" of n x n matrices over the field F is naturally identified with the vector space F An 
affine parameterized IEP requires finding within a given affine subspace of F”*" a matrix with prescribed 
spectrum. Here we will consider that the given affine subspace is n-dimensional and F = Ror F = C. 
Especially interesting is the case where the affine subspace contains only real symmetric matrices. Some 
important motivating applications, including the Sturn—Liouville problem, inverse vibration problems, 
and nuclear spectroscopy are discussed in [FNO87]. 

Most of the facts of Sections 20.8, 20.9, and 20.10 appear in [Dai98]. 


Definitions: 


An affine parameterized IEP (PIEP) has the following standard formulation: 
Given: A field F; n + 1 matrices A, Aj,..., A, € F"*"; and n elements Aj,...,A, € F. 
Find: c = (c1,...,€,)' € F" such that {Aj,... ,A,} is the spectrum of the matrix 


A(c) = A+ cA; +++: +¢,Ap- 


In particular, a PIEP(C) is a PIEP with F = C, a PIEP(R) is a PIEP with F = R, and a PIEP(RS) isa 
PIEP with F = R and with all given matrices symmetric. 


Facts: [Dai98] 
1. [Xu92] Almost all PIEP(C) are solvable. 
2. [SY86] Almost all PIEP(R) and almost all PIEP(RS) are unsolvable in the presence of multiple 
eigenvalues. 
3. All known sufficient conditions for the solvability of a PIEP(IR) or a PIEP(IRS) require that the 
eigenvalues should be sufficiently pairwise separated. An account of necessary and of sufficient 
conditions can be found in [Dai98]. 


20.9 Relevant PIEPs Which are Solvable Everywhere 


Definitions: 
Additive IEP (AIEP): Given A € C"*” and Ay,... An € C, find a diagonal matrix D € C”*” such that 
CCA LD od Mews Pk 

Multiplicative IEP (MIEP): Given B € C"*" and Ay,... ,An € C, find a diagonal matrix D € C"*" 
such that 0(BD) = {Aj,... ,An}. 

Toeplitz IEP (ToIEP): Given 41,...,, € R, finde = [c),... ,¢,]’ € R” such that [t:, | 
has spectrum {Aj,... An}. 


n . 
.._, With 
1,j=1 

fj = Cj 
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Facts: [Dai98] 


Li 


Each AIEP is a PIEP(C) with Ay = e,e; fork = 1,...,n. Each AIEP is also an IEP with prescribed 
(off-diagonal) entries. 

[Fri77] For any A € C"*” and any Aj,...,An € C, the corresponding AIEP is solvable, with 
the number of solutions not exceeding n!. Moreover, for almost all A,,... , A, there are exactly n! 
solutions. 


. Each MIEP is a PIEP(C) with A = 0 and A; = v.e, fork = 1,...,n,wherev),...,v, € C” and 


B=[v---v,] eC". 


[Fri75] Assume that all principal minors of B ¢ C"*” are nonzero. For any A;,... 5A» € C, the 
corresponding MIEP is solvable, with the number of solutions not exceeding n!. Moreover, for 
almost all A1,... , A, there are exactly n! solutions. 


. Each TolEP is a PIEP(IRS) with A = Oand A; = [ae aks where a = 1lif|i— jj +1=kand 


ij=P 
= 0 otherwise. 
[Lan94] For any A;,... , An € R the corresponding TolEP is solvable. 


alt 


20.10 Numerical Methods for PIEPs 


Facts: [Dai98] 


bk: 


For a given PIEP(RS), it is possible to order both the eigenvalues 4, < --- < A, and the eigenvalues 
Ai(c) < +--+ < An(c) of A(c). Then solving the PIEP(RS) is equivalent to solving the nonlinear 
system 


f= Gale) =A AOA =O, 


Assume that a solution c* exists and that the given eigenvalues are distinct: 


* Method 1a. Newton’s method provides a locally quadratically convergent (1.q.c.) algorithm, and 
it is usually l.q.c. even in the presence of multiple eigenvalues ([FNO87]). Each iteration in 
Newton’s method involves the solution of an eigenvalue—eigenvector problem. 


* Method 1b. A Newton-like method is given in [FNO87] . Newton’s method is modified by using 
the inverse power method to find approximate eigenvectors in each iteration. The new algorithm 
maintains l.g.c. (see [CXZ99]). 


* Method Ic. An inexact Newton-like method is given in [CCX03] . The last iterations of the inverse 
power method are truncated avoiding oversolving. The algorithm converges superlinearly, but the 
overall cost is reduced. In particular, for ToIEPs, this improved algorithm has better performance 
than specific known algorithms. 


. Fora given PIEP(R) or PIEP(C), complex eigenvalues can appear. Assume that for the correspond- 


ing PIEP a solution c* exists: 
* Method 2. In [BK81], Newton’s method is applied to solve 
f (c) = (det(A(c) — Ayn)... ,det(A(c) — AnIn))” = 0. 


The algorithm is ].q.c. and is suitable for the case of distinct eigenvalues. 


* Method 3. Newton’s method is applied in [Xu96] to solve 
f(c) = (Omin(A(e) = Ailn)s see > Omin(A(c) = Ly = 0, 


where 0;,;, denotes the smallest singular value. The algorithm is l.q.c. under mild conditions 
even when multiple eigenvalues are present. 
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* Method 4a. An 1.q.c. algorithm based on the QR decomposition theory is given in [Li92] that is 
suitable for the case of distinct eigenvalues. 


* Method 4b. Method 4a is extended in [Dai99] to the case of multiple eigenvalues for any PIEP(RS). 
The new algorithm is l.q.c. and is based on QR-like decomposition theory and least square 
techniques. Methods 4a and 4b are given in a more general context. 


3. All previous methods require starting from an initial point close to a solution in order to guarantee 
convergence. 


* Method 5. A homotopy approach has been considered for complex symmetric matrices 
(see [Chu90], [Xu93]), which in theory provides a globally convergent algorithm by which 
all solutions can be found. 
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Total positivity has been a recurring theme in linear algebra and other aspects of mathematics for the past 80 
years. Totally positive matrices, in fact, originated from studying small oscillations in mechanical systems 
[GK60], and from investigating relationships between the number of sign changes of the vectors x and Ax 
for fixed A [Sch30]. Since then this class (and the related class of sign-regular matrices) has arisen in such 
a wide range of applications (see [GM96] for an incredible compilation of interesting articles dealing with 
a long list of relevant applications of totally positive matrices) that over the years many convenient points 
of view for total positivity have been offered and later defended by many prominent mathematicians. 

After F.R. Gantmacher and M.G. Krein [GK60], totally positive matrices were seen and further developed 
in connection with spline functions, collocation matrices, and generating totally positive sequences (Polya 
frequency sequences). Then in 1968 came one of the most important and influential references in this area, 
namely the book Total Positivity by S. Karlin [Kar68]. Karlin approached total positivity by considering 
the analytic properties of totally positive functions. Along these lines, he studied totally positive kernels, 
sign-regular functions, and Polya frequency functions. Karlin also notes the importance of total positivity 
in the field of statistics. 

The next significant view point can be seen in T. Ando’s survey paper [And87]. Ando’s contribution 
was to consider a multilinear approach to this subject, namely making use of skew-symmetric products 
and Schur complements as his underlying tools. More recently, it has become clear that factorizations of 
totally positive matrices are a fruitful avenue for research on this class. Coupled with matrix factorizations, 
totally positive matrices have taken on a new combinatorial form known as Planar networks. Karlin 
and G. McGregor [KM59] were some of the pioneers of this view point on total positivity (see also 
[Bre95][Lin73]), and there has since been a revolution of sorts with many new and exciting advances and 
additional applications (e.g., positive elements in reductive Lie groups, computer-aided geometric design, 
shape-preserving designs). 

This area is not only a historically significant one in linear algebra, but it will continue to produce many 
important advances and spawn many more worthwhile applications. 
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21.1 Basic Properties 


In this section, we present many basic, yet fundamental, properties associated with the important class of 
totally positive and totally nonnegative matrices. 


Definitions: 


An m xX n real matrix A is totally nonnegative (TN) if the determinant of every square submatrix (i.e., 
minor) is nonnegative. 

Anm x n real matrix A is totally positive (TP) if every minor of A is positive. 

Ann x n matrix A is oscillatory if A is totally nonnegative and A‘ is totally positive for some integer 
k>1. 

An m x n real matrix is in double echelon form if 


(a) Each row of A has one of the following forms (* indicates a nonzero entry): 


(b) The first and last nonzero entries in rowi + 1 are not to the left of the first and last nonzero entries 
in row i, respectively (i = 1,2,...,n— 1). 


Facts: 


1. Every totally positive matrix is a totally nonnegative matrix. 

2. Suppose A is a totally nonnegative (positive) rectangular matrix. Then 
(a) A’, the transpose of A, is totally nonnegative (positive), 

(b) Ala, B] is totally nonnegative (positive) for any row index set a and column index set f. 

3. (Section 4.2) Cauchy—Binet Identity: Since any k x k minor of the product AB is a sum of products 
of k x k minors of A and B, it follows that if all the k x k minors of two n x n matrices are positive, 
then all the k x k minors of their product are positive. 

4. [GK02, p .74], [And87] The set of all totally nonnegative (positive) matrices is closed under 
multiplication. 

5. Let A be TP (TN) and D, and D, be positive diagonal matrices. Then D; AD, is TP (TN). 

6. [GKO2, p .75] If A is a square invertible totally nonnegative (or is totally positive) matrix, then 
SA7'S is totally nonnegative (positive) for S = diag(1,—1,---,+1). Hence, if A is a square 
invertible totally nonnegative matrix (or is totally positive), then the unsigned adjugate matrix of 
A (or the (n — 1)st compound of A) is totally nonnegative (positive). 

7. [And87], [Fal99] If A is a square totally nonnegative (positive) matrix, then, assuming A[a‘] is 
invertible, A/ A[a*] = Ala] — Ala, a ](A[@°])~! Ala‘, w], the Schur complement of A[a‘] in A, 
is totally nonnegative (positive), for all index sets @ based on consecutive indices. Recall that a 
denotes the complement of the set a. 

8. [And87], [Fal01], [Whi52], [Kar68, p. 98] The closure of the totally positive matrices (in the usual 
topology on R”””) is the totally nonnegative matrices. 

9. [Fal99], [JSO0] Let A = [aj,a2,...,a,] be an m x n totally nonnegative (positive) matrix whose 
it’ column is a; (i = 1,2,...,7). Suppose C denotes the set of all column vectors b for which 
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10. 


11. 


12. 


13. 


14. 
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the m x (n+ 1) matrix A = [a),... » ax, b, agya,-- 
(here k is fixed but arbitrary). Then C is a nonempty convex cone. 

[Fal99] If A is an m x n totally nonnegative (positive) matrix, then increasing the (1,1) or the 
(m,n) entries of A results in a totally nonnegative (positive) matrix. In general these are the only 
two entries in a TN matrix with this property (see [FJS00]). 

[And87] Let P denote the n x n permutation matrix induced by the permutation i > n—i+ 1, 
(1 <i <n), and suppose A is ann x n totally nonnegative (positive) matrix. Then PAP is a totally 
nonnegative (positive) matrix. 

Any irreducible tridiagonal matrix with nonzero main diagonal is in double echelon form. 

[Fal99] Let A be an m x n totally nonnegative matrix with no zero rows or columns. Then A is in 
double echelon form. 

[Rad68], [Fal99] Ann x n totally nonnegative matrix A = [a;;] isirreducible ifand only ifa;j4; > 0 
and aj41; > 0, fori = 1,2,...,n—1. 


. »@y] is a totally nonnegative (positive) matrix 


Examples: 


1. 


1 
Consider the following 3 x 3 matrix: A = | 1 
1 


. It is not difficult to check that all minors 


wn 
Noe an 


of A are positive. 

(Inverse tridiagonal matrix) From Fact 6 above, the inverse of a TN tridiagonal matrix is signature 
similar to a TN matrix. Such matrices are referred to as “single-pair” matrices in [GK02, pp. 78-80], 
are very much related to “Green’s matrices” (see [Kar68, pp. 110—112]), and are similar to matrices 
of type D found in [Mar70a]. 


. (Vandermonde matrix) Vandermonde matrices arise in the problem of determining a polynomial 


of degree at most n — 1 that interpolates n data points. Suppose that n data points (xj, y;)/_, are 
given. The goal is to construct a polynomial p(x) = ap + ajx +--+ + an—1x"—! that satisfies 
p(x;) = y; fori = 1,2,...,n, which can be expressed as 


2 
Lox Xp tee xy ao y1 
1 2 n—-1 
x2 x5 see Xy ay y2 


= (21.1) 
Vn 

The n x n coefficient matrix in (21.1) is called a Vandermonde matrix, and we denote it by 
V(x1,...,X,). The determinant of the n x n Vandermonde matrix in (21.1) is given by the formula 
is ji — xj); see [MM64, pp. 15-16]. Thus, if0 < x; < x2 <-++ < x,, then V(x),... ,x,) has 
positive entries, positive leading principal minors, and positive determinant. More generally, it is 
known [GK02, p. 111] that if0 < x; < x. <--- < x,, then V(x),...,x,) is TP. Example 1 above 
is a Vandermonde matrix. 

Let f(x) = S09 ax! be an n'" degree polynomial in x. The Routh-Hurwitz matrix is the n x n 
matrix given by 


a, a3 as a7 0 0 
a9 a2 a4 0 0 
0 a, a3 as 0 0 
A= 0 49 az a4 0 0 
0 0 0 0O an-1 O 
0 0 0 0 An—-2 an 
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A specific example of a Routh—Hurwitz matrix for an arbitrary polynomial of degree six, f(x) = 
yf aix', is given by 

a, a3 as O O 
a9 a. a4 a O 
0 a, a3 as O 
0 ao ar ag 
0 O a a3 as 
0 O ad a, a4 


oooocoeo 


A polynomial f(x) is stable if all the zeros of f (x) have negative real parts. It is proved in [Asn70] 
that f(x) is stable if and only if the Routh—Hurwitz matrix formed from f is totally nonnegative. 


. (Cauchy matrix) Ann x n matrix C = [c;;] is called a Cauchy matrix if the entries of C are given by 


1 
cit = > 
xi + Vj 

where x1, X2,...,X, and 1, y2,... » Yn are two sequences of numbers (chosen so that c;; is well- 
defined). A Cauchy matrix is totally positive if and only if0 < x) < x.) < --- < x, and 
0 < y, < yo <+++ < py ([GKO2, pp. 77-78]). 

11 1 1 

: : . 12 3 4 : . 

(Pascal matrix) Consider the 4 x 4 matrix Py = a eee The matrix P, is called the 

1 4 10 20 


symmetric 4 x 4 Pascal matrix because of its connection with Pascal’s triangle (see Example 4 of 
section 21.2 for a definition of P,, for general n). Then P, is TP, and the inverse of P, is given by 


4 —6 4 —-l1 
—6 14 —11 3 
4 —-ll 10 —3 
—l 3. 153 1 


Notice that the inverse of the 4 x 4 Pascal matrix is integral. Moreover, deleting the signs by forming 


4 6 4 1 
6 14 11 3 
-1 4: _ -_ 6 : 
SP, S, where S = diag(1,—1, 1, —1), results in the TP matrix Tat Od 24 
1 3 3 1 
Applications: 
1. (Tridiagonal matrices) When Gantmacher and Krein were studying the oscillatory properties of an 


elastic segmental continuum (no supports between the endpoints a and b) under small transverse 
oscillations, they were able to generate a system of linear equations that define the frequency of 
the oscillation (see [GK60]). The system of equations thus found can be represented in what is 
known as the influence-coefficient matrix, whose properties are analogous to those governing the 
segmental continuum. This process of obtaining the properties of the segmental continuum from 
the influence-coefficient matrix was only possible due to the inception of the theory of oscillatory 
matrices. A special case involves tridiagonal matrices (or Jacobi matrices as they were called in 
[GK02]). Tridiagonal matrices are not only interesting in their own right as a model example of 
oscillatory matrices, but they also naturally arise in studying small oscillations in certain mechanical 
systems, such as torsional oscillations of a system of disks fastened to a shaft. In [GK02, pp. 81-82] 
they prove that an irreducible tridiagonal matrix is totally nonnegative if and only if its entries are 
nonnegative and its leading principal minors are nonnegative. 
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21.2 Factorizations 


Recently, there has been renewed interest in total positivity partly motivated by the so-called “bidiagonal 
factorization,’ namely, the fact that any totally positive matrix can be factored into entry-wise nonnegative 
bidiagonal matrices. This result has proven to be a very useful and tremendously powerful property for 
this class. (See Section 1.6 for basic information on LU factorizations.) 


Definitions: 

An elementary bidiagonal matrix is an n x n matrix whose main diagonal entries are all equal to one, and 
there is, at most, one nonzero off-diagonal entry and this entry must occur on the super- or subdiagonal. 
The lower elementary bidiagonal matrix whose elements are given by 


1, ifi=j, 
Ci = Ly ifi =k, j=k-1, 
0, otherwise 


is denoted by Ex(w) = [cij] (2<k <n). 
A triangular matrix is ATP if all of its nontrivial minors are positive. (Here a trivial minor is one which 
is zero only because of the zero pattern of a triangular matrix.) 


Facts: 

1. (Ex(u))~* = Ex(-p). 

2. [Cry73] Let A bean n x n matrix. Then A is totally positive if and only if A has an LU factorization 
such that both L and U aren x n ATP matrices. 

3. [And87], [Cry76] Let A be ann x n matrix. Then A is totally nonnegative if and only if A has an 
LU factorization such that both L and U aren x n totally nonnegative matrices. 

4. [Whi52] Suppose A = [a;;] isan n x n matrix with aj), aj41,1 > 0,anday, = Ofork > j+1.Let B 
be the n x n matrix obtained from A by using row j to eliminate a j+1,1. Then A is TN ifand only if B 
is TN. Note that B is equal to E j41:(—aj41/aj1)Aj and, hence, A = (E j41(—aj41,1/aj1))'B = 
Ej41(@j4ia/aji)B. 

5. [Loe55], [GP96], [BFZ96], [FZ00], [Fal01] Let A be an n x n nonsingular totally nonnegative 
matrix. Then A can be written as 


A= (Eo (Ik))(E3(e—-1) E2(lk-2)) +++ (En (n—1) +++ E3(12) Eo) D 
(Ez (uy) E3 (ua) +++ Ey (tn-1)) +++ (Ey (u—2) E3 (ue—1))(E3 (ue), (21.2) 


where k = (5); 1i,uj > 0 for alli, j € {1,2,... ,k}; and D is a positive diagonal matrix. 
6. [Cry76] Any n x n totally nonnegative matrix A can be written as 


A=[[L°][[u™., (21.3) 


where the matrices L and U) are, respectively, lower and upper bidiagonal totally nonnegative 
matrices with at most one nonzero entry off the main diagonal. 

7. [Cry76], [RH72] If A isan n x n totally nonnegative matrix, then there exists a totally nonnegative 
matrix S and a tridiagonal totally nonnegative matrix T such that 


(a) TS=SA. 
(b) The matrices A and T have the same eigenvalues. 


Moreover, if A is nonsingular, then S is nonsingular. 
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Examples: 


1. Let Py be the matrix given in Example 6 of section 21.1. Then P, is TP, and a (unique up to a 


positive diagonal scaling) LU factorization of P, is given by 


Ci alone a 
110 0/ Jo 1 2 3 
FO hae oe a0 100 a. 

ft sca leeaveral 


Observe that the rows of L, or the columns of U, come from the rows of Pascal’s triangle (ignoring 
the zeros); hence, the name Pascal matrix (see Example 4 for a definition of P,,). 


2. The 3 x 3 Vandermonde matrix A in Example 1 of Section 21.1 can be factored as 


1 
to) [2 00 
LO): (0-1-2) 4 (21.4) 
0 1] /0 0 1 


1 0 0 1 0 0 1 0 0 
V(x], X2, x3) = 0 1 0 1 1 0 0 1 0 
0 1 1{ Jo 0 1] Jo masks 
1 0 0 1 0 0 1 x, O 1 0 0 
0 m-x, 0 0 1 x 0 1 O; JO 1 x 
0 0 (x3—22)(x3-x1) | 10 0 1] 10 O 1] Jo 0 1 


4. Consider the factorization (21.2) from Fact 5 of a 4 x 4 matrix in which all of the variables are 


equal to one. The resulting matrix is P,, which is necessarily TP. On the other hand, consider the 
n x n matrix P, = [p;;] whose first row and column entries are all ones, and for 2 < i,j <n 
let pij = Pi-1,j + pi,j—1. In fact, the relation p;; = pj-1,; + pi,j—1 implies [Fal01] that P,, can be 
written as 


1 0 


P, = E,(1)--: E2(1) F ps 


| eo --2f0. 


Hence, by induction, P,, has the factorization (21.2) in which the variables involved are all equal 
to one. Consequently, the symmetric Pascal matrix P,, is TP for all n > 1. Furthermore, since in 
general (E,(w))~' = E,(—) (Fact 1), it follows that P>' is not only signature similar to a TP 
matrix, but it is also integral. 


21.3 Recognition and Testing 


In practice, how can one determine if a given n x n matrix is TN or TP? One could calculate every minor, 
. ; 2 ; . 

but that would involve evaluating }>y_, ({)° ~ 4”/./mn determinants. Is there a smaller collection of 

minors whose nonnegativity or positivity implies the nonnegativity or positivity of all minors? 
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Definitions: 


For a = {ij,i2,... ,in} C N = {1,2,...,n}, with i) < ig <--- < ig, the dispersion of a, denoted by 
d(a), is defined to be Gn i; — 1) = ix —i, — (k — 1), with the convention that d(~) = 0, when 
a@ is a singleton. 

Ifa and £ are two contiguous index sets with |a| = |6| = k, then the minor det A[q, A] is called initial 
if a or B is {1,2,... ,k}. A minor is called a leading principal minor if it is an initial minor with both 
a= 6 =({1,2,...,k}. 

An upper right (lower left) corner minor of A is one of the form det Ala, 6] in which a consists of the 
first k (last k) and B consists of the last k (first k) indices, k = 1,2,... ,n. 


Facts: 

1. The dispersion of a set @ represents a measure of the “gaps” in the set w. In particular, observe that 
d(a) = 0 if and only if @ is a contiguous subset of N. 

2. [Fek13] (Fekete’s Criterion) An m x n matrix A is totally positive if and only if detA[a, 6B] > 0, for 
alla C {1,2,...,m}and B C {1,2,... ,n}, with |w| = |B| and d(~) = d(B) = 0. (Reduces the 
number of minors to be checked for total positivity to roughly n>.) 

3. [GP96], [FZ00] If all initial minors of A are positive, then A is TP. (Reduces the number of minors 
to be checked for total positivity to n?.) 

4. [SS95], [Fal04] Suppose that A is TN. Then A is TP ifand only if all corner minors of A are positive. 

5. [GP96] Let A € R”*” be nonsingular. 

(a) Ais TN if and only if for each k = 1,2,... ,n, 
i. det A[{1,2,...,k}] > 0. 
ii. det A[a, {1,2,...,k}] > 0, for everya C {1,2,...,n}, |a| =k. 
iii, det A[{1,2,...,k}, B] > 0, for every B C {1,2,... ,n}, |B] =k. 
(b) Ais TP if and only if for each k = 1,2... ,n, 
i. det A[a, {1,2,...,k}] > 0, for everya C {1,2,... ,n} with ja| = k, d(a) = 0. 
ii. det A[{1,2,...,k},B] > 0, for every B C {1,2,... ,n} with |B] =k, d(B) = 0. 

6. [GK02, p. 100] Ann x n totally nonnegative matrix A = [a;;] is oscillatory if and only if 
(a) A is nonsingular. 

(b) ajj41 > Oandaj+1; > 0, fori =1,2,...,n—1. 


7. [Fal04] Suppose A is an n x n invertible totally nonnegative matrix. Then A is oscillatory if 
and only if a parameter from at least one of the bidiagonal factors E, and E{ is positive, for 
each k = 2,3,...,n in the elementary bidiagonal factorization of A given in Fact 5 of 
section 21.2. 


Examples: 


1. Unfortunately, Fekete’s Criterion, Fact 2, does not hold in general if “totally positive” is replaced 
with “totally nonnegative” and “> 0” isreplaced with “> 0.” Consider the following simple example: 


1 0 2 
A=]1 0 1 . Itis not difficult to verify that every minor of A based on contiguous row and 
2 0 1 
column sets is nonnegative, but detA[{1,3}] = —3. For an invertible and irreducible example 
0 1 0 


consider A= ]0 0 1 
1 0 0 
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21.4 Spectral Properties 


Approximately 60 years ago, Gantmacher and Krein [GK60], who were originally interested in oscillation 
dynamics, undertook a careful study into the theory of totally nonnegative matrices. Of the many topics 
they considered, one was the properties of the eigenvalues of totally nonnegative matrices. 


Facts: 


1. [GKO02, pp. 86-91] Let A be ann x n oscillatory matrix. Then the eigenvalues of A are positive, real, 
and distinct. Moreover, an eigenvector x, corresponding to the k’ largest eigenvalue has exactly 
k — 1 variations in sign for k = 1,2,... ,m. Furthermore, assuming we choose the first entry of each 
eigenvector to be positive, the positions of the sign change in each successive eigenvector interlace. 
(See Preliminaries for the definition of interlace.) 

2. [And87] Let A be an n x n totally nonnegative matrix. Then the eigenvalues of A are real and 
nonnegative. 

3. [FGJ00] Let A be ann x n irreducible totally nonnegative matrix. Then the positive eigenvalues of 
A are distinct. 

4. [GK02, pp. 107-108] If A is an n x n oscillatory matrix, then the eigenvalues of A are distinct and 
strictly interlace the eigenvalues of the two principal submatrices of order n — 1 obtained from A 
by deleting the first row and column or the last row and column. If A is ann x n TN matrix, then 
nonstrict interlacing holds between the eigenvalues of A and the two principal submatrices of order 
n — 1 obtained from A by deleting the first row and column or the last row and column. 


5. [Pin98] If A is an n x n totally positive matrix with eigenvalues 4; > A, > --- > A, and 
A(k) is the (n — 1) x (nm — 1) principal submatrix obtained from A by deleting the kth row 
and column with eigenvalues 4; > 2 > +++ > Mn—1, then for j = 1,2,...,n—1,Aj-1 > 


Hj > Aj4i1, where Ag = Aj. In the usual Cauchy interlacing inequalities [MM64, p. 119] for 
positive semidefinite matrices, 2 ;_; is replaced by 4;. The nonstrict inequalities need not hold for 
TN matrices. The extreme cases (j = 1, — 1) of this interlacing result were previously proved 
in [Fri85]. 

6. [Gar82] Let n => 2 and A = [a,j] bean oscillatory matrix. Then the main diagonal entries of A are 
majorized by the eigenvalues of A. (See Preliminaries for the definition of majorization.) 


Examples: 
1 1 1 1 
: ‘ lL 2 3 4 ; 
1. Consider the 4 x 4 TP matrix Py = Le 3? we ao? Then the eigenvalues of P, are 26.305, 
1 4 #10 20 


2.203, .454, and .038, with respective eigenvectors 


.06 .53 .787 309 

.201 .64 —.163 —.723 

458 |” 392]? |—.532]}? .595 

.864 —.394 .265 —.168 
1 1 0 0 

: ; : eres 1 1 1 0 : 
2. The irreducible, singular TN (Hessenberg) matrix given by H = 1a G4 has eigenvalues 

Te ek hs ot 


equal to 3, 1, 0, 0. Notice the positive eigenvalues of H are distinct. 
3. Using the TP matrix P, in Example 1, the eigenvalues of P4({1}) are 26.213, 1.697, and .09. Observe 
that the usual Cauchy interlacing inequalities are satisfied in this case. 
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21.5 Deeper Properties 


In this section, we explore more advanced topics that are not only interesting in their own right, but 
continue to demonstrate the delicate structure of these matrices. 


Definitions: 


For a given vector ¢ = (C),C2,... 5 Cn). € R" we define two quantities associated with the number of sign 
changes of the vector c. These are: 


V~(c) — the number of sign changes in the sequence ¢),¢2,... ,C, with the zero elements discarded; 
and 

V*+(c) —the maximum number of sign changes in the sequence c1,¢2,... , Cn, Where the zero elements 
are arbitrarily assigned the values +1 and —1. 


For example, V~((1, 0, 1, -1,0, 1)7) = 2 and V*((1,0, 1, -1,0, 1)7) =4. 

We use <, < to denote the usual entry-wise partial order on matrices; i.e., for A = [aj], B = [bij] € 
R"™*", A < (<) B means ajj < (<) bjj, for alli, j. 
Let S be the signature matrix whose diagonal entries alternate in sign beginning with +. For A, B € 
R”™*", we write A < Bifandonlyif SAS < SBS andA = B ifand only if SAS < SBS, and we call this 
the “checkerboard” partial order on real matrices. 


Facts: 


1. [Sch30] Let A be an m x real matrix with m > n. If A is totally positive, then Vt(Ax) < V7 (x), 
for all nonzero x € R”. 
2. Let A= [a;;] be ann x n totally nonnegative matrix. Then: 


¢ Hadamard: [GK02, pp. 91-97], [Kot53] 
detA < Th, Qii>s 

* Fischer: [GK02, pp. 91-97], [Kot53] Let S C N = {1,2,...,n}, 
detA < detA[S] - detA[N \ S], 


* Koteljanskii: [Kot53] Let S, T C N, 
detA[S U T]- detA[S MT] < detA[S] - detA[T]. 


The above three determinantal inequalities also hold for positive semidefinite matrices. (See 
Chapter 8.5.) 
3. [FGJ03] Let a, a2, 6), and f» be subsets of {1,2,... ,m}, and let A be any TN matrix. Then 


det A[a,] det Ala] < det A[f,] det A[B>] 


if and only if each index has the same multiplicity in the multiset w Ua and the multiset 6, U fp, 
and max(|a) 9 LI, |. M Ll) > max(|B, N LI, |f2 M L|), for every contiguous (ie., d(L) = 0) 
L C {1,2,... ,n} (see [FGJ03] for other classes of principal-minor inequalities). 


4, [FJS00] Let A be an n x n totally nonnegative matrix with detA({1}) 4 0. Then A — x Ej, is totally 


nonnegative for all x € [0, aaa 


5. [FJ98] Let A be ann x n TN matrix partitioned as follows 


ae c od 


> 


where Aj, is (n — 1) x (n — 1). Suppose that rank(A;,;) = p. Then either rank({Aj,,b]) = p or 
rank[ Aj, ,c]? = p. See [FJ98] for other types of row and column inclusion results for TN matrices. 

6. [CFJ01] Let T be ann x n totally nonnegative tridiagonal matrix. Then the Hadamard product of 
T with any other TN matrix is again TN. 
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7. [CFJO1][Mar70b] The Hadamard product of any two n x n tridiagonal totally nonnegative matrices 
is again totally nonnegative. 


abc 
8. [CFJO1] LetA= |d e f]| bea3 x 3 totally nonnegative matrix. Then A has the property that 
g hi 


Ao Bis TN for all B TN if and only if 


aei+ gbf >afh-+ dbi, 
aeit+dch>afh+dbi. 


9. [CFJO1] Let A be ann x n totally nonnegative matrix with the property that A o B is TN for all B 
TN, and suppose B is any n x n totally nonnegative matrix. Then 


det(Ao B) > detB | [ aii. 


i=1 


10. [Ste91] Let A = [a;;] € R”*”, and let S,, denote the symmetric group on n symbols. If x is an 
irreducible character of S,,, then the corresponding matrix function 


n 


[ale S° x@) [[aow 


weSy i=1 


is called an immanant. For example, if x is the trivial character y(w) = 1, then the corresponding 
immanant is the permanent and, if x is sgn(w), then the immanant is the determinant. Then every 
immanant of a totally nonnegative matrix is nonnegative. 

11. [Gar96] If A,B,C € R"*", A < C < B,and A and B are TP, then det C > 0. 

12. [Gar96] If A, B,C € R"*", A < C < B, and Aand B are TP, then C is TP. 


Examples: 
1. Let 
Ls i q 
1 2 3 4 —2 
Py= 13 6 10 and letx= _s|- 
1 4 10 20 4 
Then V~(x) = 2 and Pyx = [—2, —2,5,23]7, so V*(Pyx) = 1. Hence, Schoenberg’s variation 
diminishing property (Fact 1 of section 21.5) holds in this case. 
2. Let 
1 1 0 0 1 
1 1 1 £40 1 
H= i doe and letx= - 
11141 1 


Then V~(x) = 2 and Hx = [0,—1,0,0]’, so V'(Hx) = 3. Hence, Schoenberg’s variation 
diminishing property (Fact 1 of section 21.5) does not hold in general for TN matrices. 
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3. If0 < A < Band A, B € R"" are TP, then not all matrices in the interval between A and B need 
be TP. Let 


Then A, B are TP and 


satisfies A < C < B, and C is not TP. 

4. If TP is replaced by TN, then Fact 12 no longer holds. For example, 
1 3 4 4 
* * 

0; < 0 0} < |0 
1 4 5 5 

however, both end matrices are TN while the middle matrix is not. 

5. Apolynomial f(x) is stable if all the zeros of f (x) have negative real parts, which is equivalent to the 
the Routh—Hurwitz matrix (Example 4 of Section 21.1) formed from f being totally nonnegative 
[Asn70]. Suppose f(x) = >>j_9 a;x' and g(x) = oy6 b,x! are two polynomials of degree n and 
m, respectively. Then the Hadamard product of f and g is the polynomial ( f og)(x) = S7h.y ajbix', 
where k = min(m, n). In [GW96al, it is proved that the Hadamard product of stable polynomials 
is stable. Hence, the Hadamard product of two totally nonnegative Routh—Hurwitz matrices is in 


turn a totally nonnegative matrix ([GW96a]). See also [GW96b] for a list of other subclasses of TN 
matrices that are closed under Hadamard multiplication. 


1 1 0 1 1 0 
6. Lett A = ]1 1 1] and let B = A’. Then A and B are TN, Ao B = |1 1 1), and 
Te “le od 0 1 1 


det(A o B) = —1 < 0. Thus, Ao B is not TN. 

7. (Polya matrix) Let q € (0,1). The 1 x n Polya matrix Q has its (i, ii entry equal to q~~//. Then Q 
is totally positive for all n (see [Whi52]). In fact, Q is diagonally equivalent to a TP Vandermonde 
matrix. Suppose Q represents the 3 x 3 Polya matrix. Then Q satisfies that Q o A is TN for all A 
TN whenever q € (0, ./1/}4), where sp = Lvs (the golden mean). 

8. The bidiagonal factorization (21.2) from Fact 5 of section 21.2 for TN matrices was used in [Loe55] 
to show that for each nonsingular n x n TN matrix A there is a piece-wise continuous family of 
matrices (2(f) of a special form such that the unique solution of the initial value problem 


dA(t) 
dt 


= Q(t) A(t), A(O) =I (21.5) 


has A(1) = A. Let A(t) = [a;;(t)] be a differentiable matrix-valued function of t such that A(t) is 
nonsingular and TN for all ¢ € [0,1], and A(0) = I. Then 


_ (dA(t) 
a= ( Mh (21.6) 


is called an infinitesimal element of the semigroup of nonsingular TN matrices. By (21.2) every 
nonsingular TN matrix can be obtained from the solution of the initial value problem (21.5) in 
which all Q(f) are infinitesimal elements. 

9. Ifthe main diagonal entries of a TP matrix are all ones, then it is not difficult to observe that as you 
move away from the diagonal there is a “drop off” effect in the entries of this matrix. Craven and 
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Csordas [CC98] have worked out an enticing sufficient “drop off” condition for a matrix to be TP. 
If A = [a;;] isan n x n matrix with positive entries and satisfies 
Gij4i+ij+l 2 C04i,j414i41,)> 


where cy = 4.07959562349..., then A is TP. This condition is particularly appealing for both Hankel 
and Toeplitz matrices. Recall that a Hankel matrix is an (n + 1) x (n + 1) matrix of the form 


ag a An 
a a2 An+1 
an An+1 Vee an 


So, if the positive sequence {a;} satisfies ay_jax41 = coap, then the corresponding Hankel matrix 


is TP. An (1 + 1) x (n + 1) Toeplitz matrix is of the form 


a0 a, a2 An 
a_| a0 a An-1 
a_2 a_\ a0 An-2 
A_-n GA-(n-1) A-(n-2) °*°° ao 


Hence, if the positive sequence {a;} satisfies ap > Coap—1 4x41, then the corresponding Toeplitz 
matrix is TP. 


10. Asequence a, 4),... of real numbers is called totally positive if the two-way infinite matrix given by 
ao 0 0 
a, a O 
42 a, 40 


is TP. As usual, an infinite matrix is TP all of its minors are positive. Notice that the above matrix 
is a Toeplitz matrix. Studying the functions that generate totally positive sequences was a difficult 
and important step in the area of totally positive matrices; f(x) generates the sequence do, a1,... 
if f (x) = ap +ayx +ayx? +--+ . In Aissen et al. [ASW52] (see also [Edr52]), it was shown that the 
above two-way infinite Toeplitz matrix is TP (i.e., the corresponding sequence is totally positive) 
if and only if the generating function f(x) for the sequence do, qa),... has the form 


Trai + ax) 
iaheane! ro b,x) ; 


where y,a@,, 8, > 0, and S> a, and }> B, are convergent. 


f(x) =e” 
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Linear preservers are linear maps on linear spaces of matrices that leave certain subsets, properties, relations, 
functions, etc., invariant. Linear preserver problems ask what is the general form of such maps. Describing 
the structure of such maps often gives a deeper understanding of the matrix sets, functions, or relations 
under the consideration. Some of the linear preserver problems are motivated by applications (system 
theory, quantum mechanics, etc.). 


22.1 Basic Concepts 


Definitions: 


Let V be a linear subspace of F”*”. Let f be a (scalar-valued, vector-valued, or set-valued) function on 
Vv, M asubset of V, and ~ a relation defined on V. 

A linear map ¢@ : YV > YV is called a linear preserver of function f if f(@(A)) = f(A) for every A € V. 

A linear map ¢ : V > Y preserves M if 6(M) CC M. 

The map ¢ strongly preserves M if 6(M) = M. 

A linear map ¢ : V > Y preserves the relation ~ if (A) ~ $(B) whenever A ~ B, A,B € VY. 

If for every pair A, B € V we have (A) ~ ¢(B) if and only if A ~ B, then @ strongly preserves the 
relation ~. 


Facts: 
1. If linear maps 6 : V > Vandy: V — Y both preserve M, then dy : V —> V preserves M. 
Consequently, the set of all linear transformations on V preserving M is a multiplicative semigroup. 
2. The set of all bijective linear transformations strongly preserving M is a multiplicative 
group. 
3. [BLL92, p. 41] Let M C V be an algebraic subset and ¢ : V — YV a bijective linear map satisfying 
o(M) C M. Then ¢ strongly preserves M. 
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Examples: 


1. Let R € F”*" and S € F"*". The linear map @: F"*" — F™*" defined by ¢(A) = RAS, 
A € F™*", preserves the set of all matrices of rank at most one. In general, such a map does not 
preserve this set strongly. 

2. If R and S in the previous example are invertible, then @ is a bijective linear map strongly 
preserving matrices of rank one. In fact, such a map @ strongly preserves matrices of rank k, 
k=1,...,min{m, n}. 

3. Ifm = nand R,S € F"*", then the linear map A RA'S, Ae F"™", preserves matrices of 
rank at most one. If both R and S are invertible, then ¢ strongly preserves the set of all matrices of 
rankk,l1<k<n. 

4. Assume that m = n. Let R € F"*" bean invertible matrix, c a nonzero scalar, and f : F"*" > F 
a linear functional. Then both maps A> cRAR7!+ f(A)I, Ae F™", and Ate cRATR 1+ 
f(A)I, A € F"*", strongly preserve commutativity; that is, if@ : F"*” > F"*" is any of these two 
maps, then for every pair A, B € F”*" we have #(A)o(B) = $(B)@(A) if and only if AB = BA. 

5. Let || - || be any norm on C”™”. A linear map ¢ : C”*" > C*" is called an isometry if ||@(A)|| = 
|| All, A € C’*". Thus, isometries are linear preservers of norm functions. 

6. Let 1 < k < min{m,n}. A matrix A € F*" is of rank at most k if the determinant of every 
(k +1) x (k + 1) submatrix of A is zero. Thus, the set of all m x n matrices of rank at most k is an 
algebraic subset of F’”*". 

7. The set of all nilpotent n x nm matrices is an algebraic subset of F"*". More generally, given a 
polynomial p € F[X], the set of all matrices A € F"*" satisfying p(A) = 0 is an algebraic set. 
Hence, bijective linear maps on F”*" preserving idempotent matrices, nilpotents, involutions, etc. 
preserve these sets strongly. 


22.2 Standard Forms 


Definitions: 


Let V bea linear subspace of F””*” and let P be a preserving property which makes sense for linear maps 
acting on VY (P may be the property of preserving a certain subset of V or the property of preserving a 
certain relation on Y or the property of preserving a certain function defined on V). A linear preserver 
problem corresponding to the property P is the problem of characterizing all linear (bijective) maps on 
Y satisfying this property. Very often, linear preservers have the standard forms (see the next section). 
Occasionally, there are interesting exceptional cases especially in low dimensions (see later examples). 

Let R € F™*" and S € F"*" be invertible matrices. A map @ : F™*" — F™*" is called an 
(R, S)-standard map if either ¢(A) = RAS, A € F”*", orm = nand ¢(A) = RA'S, A € F"™", 
In many cases we assume that R and S satisfy some additional assumptions. 

Let R € F"*" bean invertible matrix, c a nonzero scalar, and f : F"*" — F alinear functional. A map 
go: F"™*" —> F™*" is called an (R,c, f)-standard map if either (A) = cRAR7!+ f(A)I, Ae F™", 
or @(A) = cRA™R!4 f(A)I, Ac F™", 

When we consider linear preservers on proper subspaces V C F™*", we usually have to modify the 
notion of standard maps. Let us consider the case when V = 7, C F"*" is the subalgebra of all upper 
triangular matrices. The flip map A t> A/S, A € T,, is defined as the transposition over the antidiagonal; 
that is, Af = GA’G, where G = Ey, + Exn—-1 +... + En: (see Example 1). Standard maps on T,, are 
maps of the form 


At> RAS, AeéETh, 
Ar RA'S, AET,, 
At>cRAR!+ f(A)I, A€e Th 
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and 
At> cRASR'4 f(A), Ae€T,, 


where R and S are invertible upper triangular matrices, c is a nonzero scalar, and f is a linear functional 
on T,,. 


Facts: 


1. Every(R, S)-standard map is bijective. It strongly preserves matrices ofrankk,k = 1,... ,min{m, n}. 
Every (R,c, f)-standard map is either bijective, or its kernel is the one-dimensional subspace con- 
sisting of all scalar matrices. 

2. LetU € C"*” and V € C"*” be unitary matrices. Then a (U, V)-standard map preserves singular 
values and, hence, all functions of singular values including unitarily invariant norms. 

3. Let R € C"*” bean invertible matrix. Then an (R, R~!)-standard map on C”*” preserves spectrum, 
idempotents, nilpotents, similarity, zero products, etc. 

4. IfU € C"™" isa unitary matrix, then a (U, U*)-standard map defined on C"*” strongly preserves 
the set of all orthogonal idempotents, the set of all normal matrices, numerical range, etc. 

5. If A, B € T;, then (AB)f = BS AS, 


Examples: 
1. 
a bc djl j ig 
Oe f g 0. he fe 
00h if |0 0 e b 
000 j 0:0" 20. A 


2. Amap @: C*** > C?”? given by 


is a bijective linear map that strongly preserves commutativity but is not of a standard form. 
More generally, any bijective linear map ¢:C**? > C?*? satisfying #(I) = I strongly preserves 
commutativity [Kun99]. In higher dimensions there are no nonstandard bijective linear maps 
preserving commutativity [BLL92, p. 76]. 

3. Let W c C"*" be any linear subspace of matrices with the property that AB = BA for every pair 
A, B € W. Assume that ¢:C"*" — C"*" is a linear map whose range is contained in W. Then & 
is a nonstandard map preserving commutativity. Ifn > 1, then it does not preserve commutativity 
strongly. A map @: C’** + C?*? defined by 


(le d)-Ls 


is a concrete example of such map. 
4. Amap @:R*** > R*** given by 


a bed a b c d 

xk OK OK Ok —b a —d Cc 
1) <a 

kK OK Ok —c a —b 


x 
x 
x 
x 
| 
iy 
| 
in 
a 
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preserves the real orthogonal group, that is, @(O) is an orthogonal matrix for every orthogonal 
matrix O € R***. Note that the right-hand side of the above equation is the standard matrix 
representation of the quaternion a+bi+cj+dk. Similar constructions with matrix representations 
of complex and Cayley numbers give nonstandard linear preservers of real orthogonal group on 
R?*? and R®*®. If 6: R"*" > R"™" is a linear preserver of orthogonal group and n ¢ {2,4, 8}, 
then ¢ is a (U, V)-standard map with U and V being orthogonal matrices [LPO1, p. 601]. 

A linear map ¢ acting on 4 x 4 upper triangular matrices given by 


abcd Cr ick 28: cD 
Oe f g Oy 4 

a 
0 0h i 0 0h d 
02 EG (ia a me 


is a nonstandard bijective linear map strongly preserving the set of invertible matrices. All that is 
important in this example is that A and ¢(A) have the same diagonal entries up to a permutation. 


Standard Linear Preserver Problems 


Definitions: 


Linear 


preserver problems ask what is the general form of (bijective) linear maps on matrix spaces having 


a certain preserving property. When the general form of linear preservers under the consideration is one 
of the standard forms, we speak of a standard linear preserver problem. The following list of some most 
important standard linear preserver results is far from being complete. Many more results can be found 
in the survey [BLL92]. 


Facts: 
i 


[BLL92, Theorem 2.2], [LT92, Prop. 3] Let m, n be positive integers and @:C”*” > C”*" a linear 
map. Assume that one of the following two conditions is satisfied: 


* Letk bea positive integer, k < min{m, n}, and assume that rank (A) = k whenever rank A = k. 
* ¢ is invertible and rank f(A) = rank @(B) whenever rank A = rank B. 


Then ¢ is an (R, S)-standard map for some invertible matrices R € C’"*” and S € C"*". 


. [BLL92, p. 9] Let F bea field with more than three elements, k a positive integer, k < min{m, n}, 


and @: F™*" — F™*" an invertible linear map such that rank (A) = k whenever rank A = k. 
Then ¢ is an (R, S)-standard map for some invertible matrices R € F*” and S € F"*". 


. [BLL92, Theorem 2.6] Let 6: F*" — F?*4 bea linear map such that rank #(A) < 1 whenever 


rank A = 1. Then either 


(a) (A) = RAS for some R € F?*™ and some S € F"*? or 
(b) @(A) = RA’S for some R € F?*" and some S € F”™4 or 
(c) The range of ¢ is contained in the set of all matrices of rank at most one. 


. [BLL92, p. 10] Let F be an infinite field with characteristic 4 2, S, the space of all n x n symmetric 


matrices over F, and let k be an integer, 1 < k < n. If 6:S, — S,, is an invertible linear rank k 
preserver, then (A) = cRAR? for every A € S. Here, c is a nonzero scalar and R an invertible 
n X n matrix. 


. [BLL92, Theorem 2.9] Let F bea field with characteristic 4 2. Assume that F has more than three 


elements. Let 6: S, — S,, be a linear map such that rank @(A) < 1 whenever rank A = 1. Then 
either 
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11. 


(a) There exist an m x n matrix R anda scalar c such that $(A) = cRAR® or 


(b) The range of ¢ is contained in a linear span of some rank one m x m symmetric matrix. 


[BLL92, Theorem 2.7, Theorem 2.8, p. 25] Let @ be a real linear map acting on the real linear space 
H, ofall n x n Hermitian complex matrices. Let 7, v, 6 be nonnegative integers with wm +v+é6 =n 
and denote by G(z, v, 5) the set of all A € 7, with inertia in(A) = (z, v,6). Assume that one of 
the following is satisfied: 


* n> 3,k <n, ¢ is invertible, and rank (A) < k whenever rank A = k. 
* n> 2, the range of ¢ is not one-dimensional, and rank #( A) = 1 whenever rank A = 1. 


* The triple (z,v,5) does not belong to the set {(n, 0,0), (0, 7,0), (0,0,)}, @ is bijective, and 
(G(x, v,5)) © G(z,v, 6). 


Then there exist an invertible n x n complex matrix R anda constant c € {—1, 1} such that either 


(a) @(A) = cRAR* for every A € H,, or 

(b) 6(A) =cRA? R* for every A € Hy. 

Of course, if the third of the above conditions is satisfied, then the possibility c = —1 can occur 
only when z = v. 

[BLL92, p.76] Letn > 3andlet@: F"*" > F"*" beaninvertible linear map such that (A)¢(B) = 
(B)¢(A) whenever A and B commute. Then ¢ is an (R,c, f)-standard map for some invertible 
R € F"*", some nonzero scalar c, and some linear functional f : F"*" > F. 

[LP01, Theorem 2.3] Let p:C"*” + C"*" bea linear similarity preserving map. Then either 


(a) @ isan (R,c, f)-standard map or 
(b) @(A) = (tr AJB, AE C"*". 
Here, B, R € C"*" and R is invertible, c is a nonzero complex number, and f :C"*” > Cisa 
functional of the form f(A) = btr A, A € C"*", for some b € C. 
[BLL92, p. 77] Let @ be a real-linear unitary similarity preserving map on 7/,,. Then either 
(a) (A) = (tr A)B, A € H,, or 
(b) @(A) = cUAU* 4 b(tr A)I, A € H, or 
(c) (A) =cUATU* + b(tr A)I, A © Hy. 
Here, B is a Hermitian matrix, U is a unitary matrix, and b,c are real constants. 
[BLL92, Theorem 4.7.6] Let n > 2 and let F be an algebraically closed field of characteristic zero. 
Let p be a polynomial of degree n with at least two distinct roots. Let us write p as p(x) = x*q(x) 


with k > 0 and q(0) # 0. Assume that @: F"*" — F"*" is an invertible linear map preserving the 
set of all matrices annihilated by p. Then either 


(a) (A) =cRAR™!, Ae F"™*" or 
(b) (A) =cRA™R!, Ae F™", 

Here, R is an invertible matrix and c is a constant permuting the roots of q; that is, q(cA) = 0 
for each A € F satisfying q(A) = 0. 
[BLL92, p. 48] Let sl, C F"*" be the linear space of all trace zero matrices and ¢@:sl, > sl, 
an invertible linear map preserving the set of all nilpotent matrices. Then there exist an invertible 
matrix R € F"*" anda nonzero scalar c such that either 


(a) 6(A) =cRAR™!,A€sl, or 
(b) (A) =cRATR!, AE Sly. 
When considering linear preservers of nilpotent matrices one should observe first that the linear 


span of all nilpotent matrices is s/,, and, therefore, it is natural to confine maps under consideration 
to this subspace of codimension one. 
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[GLSOO, pp. 76, 78] Let F be an algebraically closed field of characteristic 0, m,n positive integers, 
and @: F"*" — F™*"" a linear transformation. If ¢ is nonzero and maps idempotent matrices to 
idempotent matrices, then m > n and there exist an invertible matrix R € F”*” and nonnegative 
integers ky, ky such that 1 < ky + ky, (ki + ky)n < mand 


o(A)= R(A®...PAGA'O...@ A! @O)R 


for every A € F"*", In the above block diagonal direct sum the matrix A appears k, times, A? 
appears ky times, and 0 is the zero matrix of the appropriate size (possibly absent). If p € F[X] 
is a polynomial of degree > 1 with simple zeros (each zero has multiplicity one), @ is unital and 
maps every A € F"”" satisfying p(A) = 0 into some m x m matrix annihilated by p, then ¢ is of 
the above described form with (k, + k,)n = m. 

[BLL92, Theorem 4.6.2] Let ¢:C"*” — C"*" bea linear map preserving the unitary group. Then 
g isa (U, V)-standard map for some unitary matrices U, V € C"*". 

[KH92] Let @:C"*"” + C”*" bea linear map preserving normal matrices. Then either 


(a) P(A) = cUAU* 4+ f(A)I, AE C™” or 
(b) (A) = cUA'U* 4+ f(A)I, A € C"" or 
(c) the range of ¢ is contained in the set of normal matrices. 


Here, U is a unitary matrix, c is a nonzero scalar, and f isa linear functional on C"*”. 

LPO1, p. 595] Let || - || be a unitarily invariant norm on C’”" that is not a multiple of the Frobenius 
norm defined by || Al| = \/tr (AA*). The group of linear preservers of || - || on C”*” is the group 
of all (U, V)-standard maps, where U € C”*" and V € C”*" are unitary matrices. Of course, if 
| - || is a mulitple of the Frobenious norm, then the group of linear preservers of || - || on C”™” is 
the group of all unitary operators, i-e., those linear operators @:C”*” — C”*" that preserve the 
usual inner product (A, B) = tr(AB*) on C”™*". 

BLL92, p. 63-64] Let @:C"*” — C"*" be a linear map preserving the numerical radius. Then 
either 


(a) @(A) = cUAU*, A € C"*" or 
(b) $(A) = cUATU*, Ac C"™". 


Here, U is a unitary matrix and c a complex constant with |c| = 1. 
[BLL92, Theorem 4.3.1] Let n > 2 and let 6: F"*" — F"*" be a linear map preserving the 
permanent. Then @ is an (R, S)-standard map, where R and S are each a product of a diagonal and 
a permutation matrix, and the product of the two diagonal matrices has determinant one. 
[CL98] Let @: J, — J, be a linear rank one preserver. Then either 


(a) The range of @ is the space of all matrices of the form 


* Ox * 
0 0 0 
0 0 0 


or 


(b) The range of ¢ is the space of all matrices of the form 


0 0... O x 
0 0... O x 
0 0 0 x 


or 
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(c) @(A) = RAS for some invertible R, S € T,, or 
(d) 6(A) = RASS for some invertible R, S € J,. 


Examples: 
1. Let n > 2. Then the linear map ¢: 7, > T,, defined by 


411 4\2 Gin 
0 ax a2n 
p 
0 Oo toe iy 
Ay, +422 +... t+ Ann Ay2 +423 +... 4+ An-1n +--+ Gin 
0 0 Said 0 
0 0 ae 0 


is an example of a singular preserver of rank one. 

2. The most important example of a nonstandard linear preserver problem is the problem of char- 
acterizing linear maps on n x n real or complex matrices preserving the set of positive semidef- 
inite matrices. Let Rj,...,.R;,S1,... 5S, be m x n matrices. Then the linear map @ given by 
@(A) = R,AR* +--+ R,AR* + S,ATS* +--+ + S,A7S* is a linear preserver of positive 
semidefinite matrices. Such a map is called decomposable. In general it cannot be reduced to a 
single congruence or a single congruence composed with the transposition. Moreover, there exist 
linear maps on the space of n x n matrices preserving positive semidefinite matrices that are not 
decomposable. There is no general structural result for such maps. 


22.4 Additive, Multiplicative, and Nonlinear Preservers 


Definitions: 
A map @:F”*" + F™*" is additive if (A + B) = $(A) + $(B), A,B € F™*". An additive map 
go: F™*" + F™*" having a certain preserving property is called an additive preserver. 

Amap ¢: F"*" + F"*" is multiplicative if 6(AB) = ¢(A)¢(B), A, B € F”*". A multiplicative map 
o:F"*" — F"*" having a certain preserving property is called a multiplicative preserver. 

Two matrices A, B € F"*" are said to be adjacent if rank (A — B) = 1. 

A map @: F"*" — F"*" js called a local similarity if for every A ¢ F”*” there exists an invertible 
Ra € F"*" such that 6(A) = R4ARj!. 

Let f: F — F bean automorphism of the field F. A map 6: F"*" > F"*" defined by $([ajj]) = 
[ f (a;;)] is called a ring automorphism of F"*”" induced by f. 


Facts: 


1. [BSO0] Let n > 2 and assume that ¢: F"*"” — F"*" is a surjective additive map preserving rank 
one matrices. Then there exist a pair of invertible matrices R,S € F”*" and an automorphism f 
of the field F such that ¢ is a composition of an (R, S)-standard map and a ring automorphism of 
F"*" induced by f. 

2. [GLRO3] Let SL(n,C) denote the group of all n x n complex matrices A such that det A = 1. 
A multiplicative map ¢: SL(n,C) > C"*" satisfies p(@(A)) = p(A) for every A € SL(n,C) if 
and only if there exists S € SL(n,C) such that either 
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(a) (A) = SAS™!, A € SL(n,C) or 
(b) @(A) = SAS7!, A € SL(n,C). 


Here, A denotes the matrix obtained from A by applying the complex conjugation entrywise. 
3, [PS98] Let n > 3 and let ¢:C”*" — C”*" bea continuous mapping. Then ¢ preserves spectrum, 
commutativity, and rank one matrices (no linearity, additivity, or multiplicativity is assumed) if 
and only if there exists an invertible matrix R € C”*” such that ¢ is an (R, R7!)-standard map. 
4. [BROO] Let 6:C"*” — C”*" be a spectrum preserving C!-diffeomorphism (again, we do not 
assume that ¢ is additive or multiplicative). Then @ is a local similarity. 
5. [HHW04] Let n > 2. Then 6: H,, > H,, is a bijective map such that (A) and $(B) are adjacent 
for every adjacent pair A, B € H,, if and only if there exist a nonzero real number c, an invertible 
ReC"*", and S € H, such that either 


(a) o(A) =cRAR* +S, AEH, or 
(b) (A) =cRAR*+5S,A€ Hy. 


6. [Mol01] Let n > 2 be an integer and @:H,, — H,, a bijective map such that (A) < $(B) if and 
only if A < B, A,B € H, (here, A < B if and only if B — A is a positive semidefinite matrix). 
Then there exist an invertible R € C”*" and S € H,, such that either 


(a) (A) = RAR* +S, A€H, or 
(b) (A) = RAR* +S, AE Hn. 


This result has an infinite-dimensional analog important in quantum mechanics. In the language 
of quantum mechanics, the relation A < B means thatthe expected value of the bounded observable 
A in any state is less than or equal to the expected value of B in the same state. 


Examples: 


1. We define a mapping @:C"*” — C”*" in the following way. For a diagonal matrix A with distinct 
diagonal entries, we define @(A) to be the diagonal matrix obtained from A by interchanging the 
first two diagonal elements. Otherwise, let @(A) be equal to A. Clearly, @ is a bijective mapping 
preserving spectrum, rank, and commutativity in both directions. This shows that the continuity 
assumption is indispensable in Fact 3 above. 

2. Let @:C"*” — C"*" be a map defined by #(0) = Ey, (E12) = 0, and ¢(A) = A for all 
Ae C"*" \ {0, Ey}. Then ¢ is a bijective spectrum preserving map that is not a local similarity. 
More generally, we can decompose C”*” into the disjoint union of the classes of matrices having 
the same spectrum and then any bijection leaving each of this classes invariant preserves spectrum. 
Thus, the assumption on differentiability is essential in Fact 4 above. 
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In this chapter, we present some results on matrices over integral domains, which extend the well-known 
results for matrices over the fields discussed in Chapter 1 of this book. The general theory of linear algebra 
over commutative rings is extensively studied in the book [McD84]. It is mostly intended for readers with 
a thorough training in ring theory. The aim of this chapter is to give a brief survey of notions and facts 
about matrices over classical domains that come up in applications. Namely over the ring of integers, the 
ring of polynomials over the field, the ring of analytic functions in one variable on an open connected set, 
and germs of analytic functions in one variable at the origin. The last section of this chapter is devoted to 
the notion of strict equivalence of pencils. 

Most of the results in this chapter are well known to the experts. A few new results are taken from the 
book in progress [Frixx], which are mostly contained in the preprint [Fri81]. 


23.1 Certain Integral Domains 


Definitions: 
A commutative ring without zero divisors and containing identity 1 is an integral domain and denoted 
by D. 

The quotient field F of a given integral domain D is formed by the set of equivalence classes of all 
quotients eo b # 0, where 5 = 4 if and only if ad = bc, such that 


ac ad + bc ac ac 


T — > = ) b,d 0. 

bod bd bd bd a 

For x = [x,,... 5 Xn]? € D",a = [ay,... ,a,]7 € Z% we define x” = xf" --- x% and |a| = 77, lail. 
D[x] = D[x,... , x, ] isthe ring ofall polynomials p(x) = p(x1,...,x,) inn variables with coefficients 


inD: 
p(x) = = Ay x”. 
la|<m 
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The total degree, or simply the degree of p(x) 4 0, denoted by deg p, is the maximum m € Z4 such 
that there exists a, # 0 such that |a| = m. (deg 0 = —oo.) 

A polynomial p is homogeneous if a, = 0 for all |a| < deg p. 

A polynomial p(x) = S7j_9 a;x' € D[x] is monic if a, = 1. 

F (x) denotes the quotient field of F [x], and is the field of rational functions over F in n variables. 

Let 2 Cc C” be a nonempty path-connected set. Then H(Q2) denotes the ring of analytic functions 
f(z), such that for each ¢ € © there exists an open neighborhood O(¢, f) of ¢ such that f is analytic 
on O(f,¢). The addition and the product of functions are given by the standard identities: (f + g)(¢) = 
f(O)+8(), (feo) = f(o)g(f). IfQ is an open set, we assume that f is defined only on . If Q consists 
of one point ¢, then H; stands for H({¢}). 

Denote by M(Q), M; the quotient fields of H(Q), H; respectively. 

For a,d € D, d divides a (or d is a divisor of a), denoted by d|a, ifa = db for some b € D. 

a € Dis unit ifal1. 

a,b € Dare associates, denoted by a = J, if a|b and bja. Denote {{a}} = {b €D: b=a}. 

The associates of a € D and the units are called improper divisors of a. 

a € Dis irreducible if it is not a unit and every divisor of a is improper. 

A nonzero, nonunit element p € D is prime if for any a,b € D, p|ab implies pla or p|b. 

Leta),...,a, € D. Assume first that notall ofa),... ,a, are equal to zero. Anelementd € Disa greatest 
common divisor (g.c.d) of a1,... , a, ifd|a; fori = 1,... ,n, and for any d’ such that d’|a;,i = 1,... ,n, 
d'|d. Denote by (a),... ,a,) any g.c.d. of a1,... ,a@,. Then {{(a1,... ,@_)}} is the equivalence class of all 
g.c.d. of aj,...,@,. Fora, =... =a, = 0, we define 0 to be the g.c.d. of aj,... dy ie. (a1,... 5 an) = 0. 

a,,... 5a, € Dare coprime if {{(a),... 5a,)}} = {{1}}. 

I C Dis an ideal if for any a,b € I and p,q € D the element pa + qb belongs to I. 

An ideal I #4 Dis prime if ab € I implies that either a or b is in I. 

An ideal I 4 D is maximal if the only ideals that contain I are J and D. 

An ideal I is finitely generated if there exists k elements (generators) p,,..., px € I such that any 
i € I is ofthe formi = a, p; +--- + a, px for some aj,... , ax € D. 

An ideal is principal ideal if it is generated by one element p. 
D is a greatest common divisor domain (GCDD), denoted by D,, if any two elements in D have a g.c.d. 
D is a unique factorization domain (UFD), denoted by D,,, if any nonzero, nonunit element a can be 


oO 


factored as a product of irreducible elements a = p, --- p;, and this factorization is unique within order 
and unit factors. 

D is a principal ideal domain (PID), denoted D,, if any ideal of D is principal. 

D is a Euclidean domain (ED), denoted D,, if there exists a function d : D\{0} — Z, such that: 


for alla,be€D, ab40, d(a) < d(ab); 
for any a,b € D, ab # 0, there exists t,r € D such that 
a=tb-+r, where either r = 0 or d(r) < d(b). 


It is convenient to define d(0) = oo. Let a),a. € D, and assume that oo > d(a,) > d(az). Euclid’s 
algorithm consists of a sequence 4j,... ,a%41, where (a,...a,) 4 0, which is defined recursively as 
follows: 


Q; = tAj4, +Qj42,  Aj42 = 0 or d(aj42) < d(aj4,) for i=1,...k—1. 


[Hel43], [Kap49] A GCDD D is an elementary divisor domain (EDD), denoted by D,g, if for any three 
elements a,b,c € D there exists p,q,x,y € D such that (a,b,c) = (px)a + (py)b + (qy)c. 

A GCDD Disa Bezout domain (BD), denoted by Dy, iffor any two elements a,b € D (a,b) = pa+qb, 
for some p,q € D. 

p(x) = y5 ajx"—' € Z[x],a9 £0,m > Lis primitive if 1 is a g.c.d. of ag,... 54m. 

For m €N, the set of integers modulo m is denoted by Z,,,. 
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Facts: 


Most of the facts about domains can be found in [ZS58] and [DF04]. More special results and references 
on the elementary divisor domains and rings are in [McD84]. The standard results on the domains of 
analytic functions can be found in [GR65]. More special results on analytic functions in one complex 
variable are in [Rud74]. 


1. Any integral domain satisfies cancellation laws: ifab = ac or ba = ca anda 4 0, thenb = c. 
2. An integral domain such that any nonzero element is unit is a field F, and any field is an integral 
domain in which any nonzero element is unit. 
3. A finite integral domain is a field. 
4, D[x] is an integral domain. 
5. H(Q) is an integral domain. 
6. Any prime element in D is irreducible. 
7. Ina UED, any irreducible element is prime. This is not true in all integral domains. 
8. Let D be a UED. Then D[x] is a UFD. Hence, D[x] is a UFD. 
9. Let a},a, € D, and assume that co > d(a,) > d(az). Then Euclid’s algorithm terminates in a 
finite number of steps, i.e., there exists k > 3 such that a; 4 0,...,a, #0 and az4, = 0. Hence, 
ay = (41,42). 
10. An ED isa PID. 
11. A PID is an EDD. 
12. A PID isa UFD. 
13. An EDD isa BD. 
14. A BD isa GCDD. 
15. A UFD isa GCDD. 
16. The converses of Facts 10, 11, 12, 14, 15 are false (see Facts 28, 27, 21, 22). 
17. [DF04, Chap. 8] An integral domain that is both a BD and a UFD is a PID. 
18. An integral domain is a Bezout domain if and only if any finitely generated ideal is principal. 
19. Zis an ED with d(a) = |al. 
20. Let p,q € Z[x] be primitive polynomials. Then pq is primitive. 
21. F[x] is an ED with d(p) = the degree of a nonzero polynomial. Hence, F [x1,... ;x,] is a UFD. 
But F[x),...,x,] isnot a PID forn > 2. 
22. Z[x1,... Xm], Flx1,... Xn], and H(Q) (for a connected set Q C C”) are GCDDs, but for m > 1 
and n > 2 these domains are not B Ds. 
23. [Frixx] (See Example 17 below.) Let Q C C be an open connected set. Then for a, b € H(&2) there 
exists p € H(Q) such that (a,b) = pa+b. 
24. H;, ¢ € C,isa UFD. 
25. If 2 C C" is a connected open set, then H({2) is not a UFD. (For n = 1 there is no prime 
factorization of an analytic function f € H(Q) with an infinite countable number of zeros.) 
26. Let 2 C C be a compact connected set. Then H(Q) is an ED. Here, d(a) is the number of zeros of 
a nonzero function a € H(2) counted with their multiplicities. 
27. [Frixx] If Q C C is an open connected set, then H(Q) is an EDD. (See Example 17.) As H(Q) 
is not a UFD, it follows that H(Q) is not a PID. (Contrary to [McD84, Exc. II.E.10 (b), 
p. 144].) 
28. [DF04, Chap. 8] Z[(1 + /—19)/2] is a PID that is not an ED. 
Examples: 
1. {1,—1} is the set of units in Z. A g.c.d. of aj,... ,a, € Zis uniquely normalized by the condition 
(a,,...,ax) = 0. 
2. A positive integer p € Z is irreducible if and only if p is prime. 
3. Zm is an integral domain and, hence, a field with m elements if and only if p is a prime. 
4. ZD I isa prime ideal if and only if all elements of I are divisible by some prime p. 
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15. 


16. 


17. 


18. 
19. 
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{1, —1} is the set of units in Z[x]. A g.c.d. of pi,..., px € Z[x], is uniquely normalized by the 
condition (pi,... 5 Pe) = doj29 ax and ay > 0. 
Any prime element in p(x) € Z[x],deg p > 1, is a primitive polynomial. 


Let p(x) = 2x+3, q(x) = 5x—3 € Z[x]. Clearly p, q are primitive polynomials and (p(x), q(x))=1. 
However, 1 cannot be expressed as 1 = a(x) p(x) + b(x)q(x), where a(x), b(x) € Z[x]. Indeed, 
if this was possible, then 1 = a(0)p(0) + b(0)q(0) = 3(a(0) — b(0)), which is impossible for 
a(0), b(0) € Z. Hence, Z[x] is not BD. 

The field of quotients of Z is the field of rational numbers Q. 

Let p(x), q(x) € Z[x] be two nonzero polynomials. Let (p(x), q(x)) be the g.c.d of p,q in Z[x]. 
Use the fact that p(x), q(x) € Q[x] to deduce that there exists a positive integer mand a(x), b(x) € 
Z{x] such that a(x) p(x) + b(x)q(x) = m(p(x), q(x)). Furthermore, if c(x) p(x) + d(x)q(x) = 
I( p(x), q(x)) for some c(x), d(x) € Z[x] and 0 41 € Z, then mll. 


. The set of real numbers R and the set of complex numbers C are fields. 
. Ag.c.d. of aj,...,a, € F[x] is uniquely normalized by the condition (p),... , py) is a monic 


polynomial. 


. A linear polynomial in D[x] is irreducible. 
. Let Q Cc C bea connected set. Then each irreducible element of H(Q) is an associate of z — ¢ for 


some ¢ € Q. 


. For ¢ € C Hg, every irreducible element is of the form a(z — ¢) forsome 0 4 a € C. Ag.cd. 


of a),... ,a% € Hy is uniquely normalized by the condition (a),... ,a,) = (z — ¢)” for some 

nonnegative integer m. 

In H(Q), the set of functions which vanishes on a prescribed set U € Q, i.e., 

I(U):={f €H(Q):  f(¢) =0, ¢ € U}, 

is an ideal. 

Let Q be an open connected set in C. [Rud74, Theorems 15.11, 15.13] implies the following: 

° I(U) ¥ {0} if and only if U is a countable set, with no accumulations points in Q. 

* Let U bea countable subset of Q with no accumulation points in 2. Assume that for each ¢ € U 
one is given a nonnegative integer m(¢) and m(¢) + 1 complex numbers wo,,... , Wm(c),¢- Then 
there exists f € H(Q) such that f(O) =nlwyc,n =0,... ,m(¢), forall¢ € U. Furthermore, 
if all w,,¢ = 0, then there exists g € H(Q) such that all zeros of g are in U and g has a zero of 
order m(¢) + lateach¢é € U. 

° Let a,b € H(Q),ab # 0. Then there exists f € H(Q) such that a = cf,b = df, where 
c,d € H(Q) and c,d do not have a common zero in Q. 

* Letc,d € H(Q) and assume that c, d do not have a common zero in Q. Let U be the zero set of c 
in Q, and denote by m(¢) > 1 the multiplicity of the zero ¢ € U ofc. Then there exists g € H(Q) 
such that (e8)(¢) = d‘™(¢) for n = 0,...,m(£), for all ¢ € U. Hence, p = and € H(Q) 
and e§ = pe + d isa unit in H(Q). 

* For a,b € H(Q) there exists p € H(Q) such (a,b) = pa+b. 

* For a,b,c € H(Q) one has (a,b,c) = p(a,b) +c = p(xa +b) +c. Hence, H(Q) is EDD. 

Let I C C[x, y] be the ideal given by given by the condition p(0,0) = 0. Then I is generated by x 

and y, and (x, y) = 1. J is not principal and C[x, y] is not BD. 

D[x, y] is not BD. 

Z{,/—5] is an integral domain that is not a GCDD, since 2 + 2,/—5 and 6 have no greatest common 

divisor. This can be seen by using the norm N(a + b./—5) = a? + 5b’. 


23.2 Equivalence of Matrices 


In this section, we introduce matrices over an integral domain. Since any domain D can be viewed as a 
subset of its quotient field F, the notion of determinant, minor, rank, and adjugate in Chapters 1, 2, and 
4 can be applied to these matrices. It is an interesting problem to determine whether one given matrix can 
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be transformed to another by left multiplication, right multiplication, or multiplication on both sides, 
using only matrices invertible with the domain. These are equivalence relations and the problem is to 
characterize left (row) equivalence classes, right (columns) equivalence classes, and equivalence classes 
in D”*", For BD, the left equivalence classes are characterized by their Hermite normal form, which are 
attributed to Hermite. For EDD, the equivalence classes are characterized by their Smith normal form. 


Definitions: 
For a set S, denote by S’”*” the set of all m x n matrices A = [age where each aj; € S. 

For positive integers p < q, denote by Q,p,, the set of all subsets {i,... ,ip} C {1,2,...q} of cardinality 
p, where we assume that 1 <i, <...<ip <q. 


U € D"*" is D-invertible (unimodular), if there exists V € D’™” such that UV = VU = Ij. 

GL(n, D) denotes the group of D-invertible matrices in D’*”. 

Let A, B € D”*". Then A and B are column equivalent, row equivalent, and equivalent if the following 
conditions hold respectively: 


B=AP_ forsome P €GL(n,D) (A~, B), 
B=QA _ forsome QE GL(m,D) (A~, B), 
B=QAP _ forsome P € GL(n,D), Qe GL(m,D) (A~ B). 


For Aé br", let 
La, A) = g.c.d. {det A[a, 6], O€ Qin} ae Qi 


v(B, A) = g.c.d. {det Aldg, B], wv) € Qik} B € Qin 
5x (A) = g.c.d. {det A[¢,6], 6 € Qums O € Quen}. 


6;( A) is the k-th determinant invariant of A. 
For A€é Dens 


6;(A) 


ij(A) = 5A)’ 


j=l,...,rank A, (89(A) = 1), 


ij(A)=0 for rank A < j < min(m,n), 


are called the invariant factors of A. i;(A) is a trivial factor if i;(A) is unit in D,. We adopt the normal- 
ization i;(A) = 1 for any trivial factor of A. For D = Z, Z[x], F [x], we adopt the normalizations given 
in the previous section in the Examples 1, 6, and 12, respectively. 

Assume that D[x] is a GCDD. Then the invariant factors of A € D[x]”™" are also called invariant 
polynomials. 

D = [d;;] € D”*" isa diagonal matrix if d;; = 0 for alli # j. The entries dy,... , dee, € = min(m,n), 
are called the diagonal entries of D. D is denoted as D = diag (dj),... , dee) € D”™*". 

Denote by II, C GL(n, D) the group of n x n permutation matrices. 


AnD-invertible matrix U € GL(n, D) is simple ifthere exists P, Q € TI, suchthat U = P : i | Q, 
n—2 
a £B : . . : 
where V = ys € GL(2, D), ie., a5 — By is D-invertible. 
U is elementary if U is of the above form and V = ' € GL(2, D), ie., a, 6 are invertible. 


For A € D”*", the following row (column) operations are elementary row operations: 


(a) Interchange any two rows (columns) of A. 
(b) Multiply row (column) i by an invertible element a. 
(c) Add to row (column) j b times row (column) i (i # 7). 


23-6 Handbook of Linear Algebra 


For A € D”*", the following row (column) operations are simple row operations: 


(d) Replace row (column) i by a times row (column) i plus b times row (column) j, and row (column) 
j by c times row (column) i plus d times row (column) j, wherei # j and ad — bc is invertible 
in D. 


B = [b,j] € D”*" is in Hermite normal form if the following conditions hold. Let r = rank B. First, 
the i-th row of B is a nonzero row if and only ifi < r. Second, let b;,, be the first nonzero entry in the 
i-th row fori = 1,...,r. Then 1 < nm; < m <::- <n, <n. 

Be bes" is in Smith normal form if B is a diagonal matrix B = diag (bj,... ,b,,0,... ,0), bj 4 0, 
fori =1,...,r and b;_,|b; fori =2,...,r. 


Facts: 


Most of the results of this section can be found in [McD84]. Some special results of this section are given 
in [Fri81] and [Frixx]. For information about equivalence over fields, see Chapter 1 and Chapter 2. 


1. The cardinality of Qp,q is +) : 
2. If U is D-invertible then det U is a unit in D. Conversely, if det U is a unit then U is D-invertible, 
and its inverse U~! is given by U! = (det U)~!adj U. 
3. For A € D”*", the rank of A is the maximal size of the nonvanishing minor. (The rank of zero 
matrix is 0.) 
4. Column equivalence, row equivalence, and equivalence of matrices are equivalence relations in 
p”"*", 
5. Forany A,B € D”*",onehasA~, B <=> A! ~, B! . Hence, it is enough to consider the row 
equivalence relation. 
6. For A, Be Dees the Cauchy—Binet formula (Chapter 4) yields 
(a, A) = wa, B) foralla € Qum if A~, B, 
v(B,A)=v(f,B) forallB € Qk, if A~, B, 
64(A) = 6,(B) if A~ B, 
fork =1,...,min(m,n). 
7. Any elementary matrix is a simple matrix, but not conversely. 
8. The elementary row and column operations can be carried out by multiplications by A by suitable 
elementary matrices from the left and the right, respectively. 
9. The simple row and column operations are carried out by multiplications by A by suitable simple 
matrices U from the left and right, respectively. 
10. LetD, bea Bezout domain, A € Dj’*”, rank A = r.Then Aisrowequivalentto B = [b;;] € D,’*", 
in a Hermite normal form, which satisfies the following conditions. 
Let bj, be the first nonzero entry in the i-th row fori = 1,...,r. Then 1 < ny <m <-+-: < 
n, < nare uniquely determined and the elements bj,,, i = 1,...,7 are uniquely determined, up 
to units, by the conditions 


V((m,,... 51; )> A) = Din, +++ Din, od ea 
v(a, A) = 0, @ € Qin, i=l... sr. 


The elements bj,,, j = 1,...,i — 1 are then successively uniquely determined up to the 
addition of arbitrary multiples of bj,,. The remaining elements bj, are now uniquely determined. 
The D-invertible matrix Q, such that B = QA, can be given by a finite product of simple matrices. 

If b;,, in the Hermite normal form is invertible, we assume the normalization conditions b;,, = 1 
and bj,, = 0 fori < j. 

11. For Euclidean domains, we assume normalization conditions either b j,, = 0 or d(Djn,) < d(bin;) 
for j < i. Then for any A € D?’*", in a Hermite normal form B = QA, Q € GL,,(D,) Q isa 
product of a finite elementary matrices. 
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12. 


13. 


14. 


15. 


16. 
17. 
18. 


19. 


U € GL(n, D,) is a finite product of elementary D-invertible matrices. 

For Z, we assume the normalization b;,, => 1 and 0 < bjn,; < bin, for j < i. For F [x], we assume 
that b;,, is a monic polynomial and deg bj,, < deg bj,, for j < i. Then for D, = Z, F [x], any 
A € D”"*" has a unique Hermite normal form. 

A, B € D, are row equivalent if and only if A and B are row equivalent to the same Hermite normal 
form. 

Aeé F™*" can be brought to its unique Hermite normal form, called the reduced row echelon form 
(RREF), 


bin, = 1, Bjn, = 90, fol,...,i-1, i= 1,...,r =rank A, 


by a finite number of elementary row operations. Hence, A, B € F™™” are row equivalent if and 
only ifr = rank A= rank B and they have the same RREF. (See Chapter 1.) 

For A€ brs” and 1 < p <q < min(m,n), 5,(A)|5,(A). 

For Aé bres", i;-1(A)|i;(A) for j = 2,...,rank A. 

Any 0 4 A € DX" is equivalent to its Smith normal form B = diag (i;(A),... ,i,(A),0,... ,0), 
where r = rank A andij(A),... ,i,(A) are the invariants factors of A. 

A,B € DX" are equivalent if and only if A and B have the same rank and the same invariant 
factors. 


Examples: 


1, 


4. 


. Let A= 


Let A = 0 : B= : : € D?*? be two Hermite normal forms. It is straightforward to 
show that A ~, B if and only if a = b. Assume that D is a BD and let a # 0. Then rank A = 
1, v((1), A) = 1, {{v((2), A)}} = {{a}}, v((1, 2), A) = 0. If D has other units than 1, it follows that 
v(B, A) for all 8 € Qx2,k = 1,2 do not determine the row equivalence class of A. 

: : € De ** Then there exists u,v € D, such that ua + vb = (a,b) = v((1), A). 
If (a,b) # 0, then 1 = (u,v). Ifa = b = 0, choose u = 1,v = 0. Hence, there exists x, y € Dy 


(a,b) c’ 


such that yu — xv = 1. Thus V = ‘ ; € GL(2,D,) and VA = b! A Clearly 


0 (a,b) c’ 
1 0 f 

This construction is easily extended to obtain a Hermite normal form for any A € Dj’*", using 
simple row operations. 


b' = xa + yb = (a,b)e. Hence, VA= is a Hermite normal form of A. 


. Let A € D2*? as in the previous example. Assume that ab # 0. Change the two rows of A if 


needed to assume that d(a) < d(b). Let aj = b, az = a and do the first step of Euclid’s algorithm: 


a, = tia, + 43, where a3 = 0 or d(a3) < d(a2). Let V = € GL(2, D,) be an elementary 


te. il 


matrix. Then Ay = VA = i i . If a3 = 0, then A; has a Hermite normal form. If a; 4 0, 
3 


continue as above. Since Euclid’s algorithm terminates after a finite number of steps, it follows 
that A can be put into Hermite normal form by a finite number of elementary row operations. 
This statement holds similarly in the case ab = 0. This construction is easily extended to obtain a 
Hermite normal form for any A € D?"*" using elementary row operations. 

. a b . ; 
Assume that D isa BD and let A = ; € D’*?. Note that 6;(A) = (a, b,c). If A is equivalent 


to a Smith normal form then there exists V, U € GL(2, D) such that VAU = 


(a,b,c) * 
* 


ral 
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Assume that V = ; s Us y : . Then there exist p,q,x,y € D such that (px)a + 
(py)b + (qy)c = (a,b,c). Thus, if each A € D?*? is equivalent to Smith normal form in D?*?, 
then it follows that D is an EDD. 

Conversely, suppose that D is an EDD. Then D is also a BD. Let A € D**?. First, bring A to an 
upper triangular Hermite normal form using simple row operations: A} = WA = : »,We 
GL(2,D). Note that 6;(A) = 6,(A;) = (a, b,c). Since D is an EDD, there exist p,q,x, y € D such 
that (px)a+(py)b+(qy)c = (a,b,c). If (a,b,c) 4 (0,0, 0), then (p,q) = (x, y) = 1. Otherwise 
A = A, = Oand we are done. Hence, there exist p,q, x,y such that pp — qq = xx — yy = 1. 
Vs oe Se ara a Se 

q Pp y x §21 $22 
deduce that 5, (A) divides gj. and gy). Apply appropriate elementary row and column operations to 
deduce that A is equivalent to a diagonal matrix C = diag (i;(A), d)). As 62(C) = i,(A)d) = 62(A), 
we see that C is has Smith normal form. 

These arguments are easily extended to obtain a Smith normal form for any A € D”*", using 


. Since 6,(G) = 6,(A), we 


simple row and column operations. 
5. The converse of Fact 6 is false, as can be seen by considering 


oe LOD e zpc}*2. w({1), A) = w({1},B) = 1, w({2}, A) = (2), B) = 1, 


im x x|? Be 0 0 
w({1, 2}, A) = w({1, 2}, B) = 0, but there does not exist a D-invertible P such that PA = B. 

6. Let D be an integral domain and assume that p(x) = x” + oi", a;x"-? € D[x] is a monic 
polynomial of degree m > 2. Let C(p) € D”*” be the companion matrix (see Chapter 4). Then 
det (xIm—C(p)) = p(x). Assume that D[x] isa GCDD. Let C(p)(x) = xIm —C(p) € D[x]”™*”. 
By deleting the last column and first row, we obtain a triangular (m— 1) x (m— 1) submatrix with 
—1s on the diagonal, so it follows that 6;(C(p)(x)) = 1 fori = 1,... ,m— 1. Hence, the invariant 
factors of C(p)(x) are i;(C(p)(x)) =... = im-1(C(p)(x)) = land in(C(p)(x)) = p(x). IfD is 
a field, then C(p)(x) is equivalent over D[x] to diag (1,... , 1, p(x)). 


23.3 Linear Equations over Bezout Domains 


Definitions: 


M is a D-module if M is an additive group with respect to the operation + and M admits a multiplication 
by a scalar a € D, i-e., there exists a mapping D x M — M which satisfies the standard distribution 
properties and lv = v for all v € M (the latter requirement sometimes results in the module being called 
unital). (For a field F, a module M is a vector space over F..) 

M is finitely generated if there exist v|,... ,V, € M so that every v € M is a linear combination of 
V1>.-+- 5 Vp, Over D, ie. v= ayvj +... + anv, for some aj,... 54, € D. v),... ,V, is a basis of M if every 
v can be written as a unique linear combination of vj,... ,v,. dim M = n means that M has a basis of n 
elements. 

N C Misa D-submodule of M if N is closed under the addition and multiplication by scalars. 

D"(= D"*') is a D-module. It has a standard basis e; fori = 1,... ,n, where e; is the i-th column of 
the identity matrix I,,. 

For any A € D’"*", the range of A, denoted by range(A), is the set of all linear combinations of the 
columns of A. 

The kernel of A, denoted by ker(A), is the set of all solutions to the homogeneous equation Ax = 0. 

Consider a system of m linear equations in n unknowns: 
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jal ajjXj = bj, i =1,...,m, where ajj,b; € D fori = 1,...,m, j = 1,...,n. In matrix 

notation this system is Ax = b, where A = [a;;] € D””", x = [x),... ,xn]' €D", b= [b,,...,b,]" € 
D™. Aand[A,b] € D”* + are called the coefficient matrix and the augmented matrix, respectively. 

Let A € Hj’*". Then A = A(z) = [aij (2) jj and A(z) has the McLaurin expansion A(z) = 
ae Apz*, where Ay € C”%",k = 0,.... Here, each a;;(z) has convergent McLaurin series for |z| < 
R(A) for some R(A) > 0. 

The invariant factors of A are called the local invariant polynomials of A, which are normalized to be 
of the form ix(A) = z'* for 0 < i; <i2 <... <i;, wherer = rank A. 

The integer i, is the index of A and is denoted by 7 = n(A). For a nonnegative integer p, denote by 
Ky = K (A) the number of local invariant polynomials of A whose degree is equal to p. 


Facts: 


For modules, see [ZS58], [DF04], and [McD84]. The solvability of linear systems over EDD can be traced 
to [Hel43] and [Kap49]. The results for BD can be found in [Fri81] and [Frixx]. The results for Ho are 
given in [Fri80]. The general theory of solvability of the systems of equations over commutative rings is 
discussed in [McD84, Exc. I.G.7—-I.G.8]. (See Chapter 1 for information about solvability over fields.) 


1. The system Ax = b is solvable over a Bezout domain Dy if and only ifr = rank A = rank [A,b] 
and 6,(A) = 6,([A,b]), which is equivalent to the statement that A and [A,b] have the same 
set of invariant factors, up to invertible elements. For a field F, this result reduces to the equality 
rank A = rank [A,b]. 

2. For A € Dj’*", range A and ker A are modules in D}’ and D} having finite bases with rank A and 
null A elements, respectively. Moreover, the basis of ker A can be completed to a basis of Dj. 

3. For A, B € Hy"*", let C(z) = A(z) + z**!B(z), where k is a nonnegative integer. Then A and C 
have the same local invariant polynomials up to degree k. Moreover, if k is equal to the index of A, 
and A and C have the same rank, then A is equivalent to C. 

4. Consider a system of linear equations over Hp A(z)u(z) = b(z), where A(z) € Hj’*” and b(z) = 
Sey bez* € Hi", by € C", k = 0, .... Look for the power series solution u(z) = 7729 uxz*, 
where uz € C”, k =0,.... Then He Az_juj = by, fork = 0,... . This system is solvable for 
k=0,...,q € Z, if and only if A(z) and [ A(z), b(z)] have the same local invariant polynomials 
up to degree q. 

5. Suppose that A(z)u(z) = b(z) is solvable over Ho. Let q = n(A) and suppose that ug,... , Uy 
satisfies the system of equations, given in the previous fact, for k = 0,... ,q. Then there exists a 
solution u(z) € Hj satisfying u(0) = uo. 

6. Letq € Z, and W, C C” be the subspace of all vectors wo such that Wo,... ,W, is a solution to 
the homogenous system ee Ax—jwj = 0, fork =0,...,q.Thendim W, =n — 4 Kj (A). 
In particular, for 7 = n(A) and any wo € W,, there exists w(z) € Hj such that A(z)w(z) = 0, 
w(0) = Wo. 


23.4 Strict Equivalence of Pencils 


Definitions: 
A matrix A(x) € D[x]'"*” is a pencil if A(x) = Ap + x Aj, Ao, Ay € D”*”. 

A pencil A(x) is regular ifm = nand det A(x) is not the zero polynomial. Otherwise A(x) is a singular 
pencil. 

Associate with a pencil A(x) = Ap +xA, € D[x]”*”" the homogeneous pencil A(x9, x1) = xp Ao + 
x, A, € D[xo, x, ]'"*". 

Two pencils A(x), B(x) € D[x]'"*" are strictly equivalent, denoted by A(x)~B(x),if B(x) = QA(x)P 
for some P € GL,(D), Q € GL,,(D). Similarly, two homogeneous pencils A(x, x1), B(xo, x1) 
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€ D[xo,x;]”*" are strictly equivalent, denoted by A(x. X1)~ B(x X1)) if B(xo, x1) = QA(x,x1)P 
for some P € GL, (D), Q € GL,,(D). 

For a UFD D, let 5: (xo,x1), ix(xo, x1) be the invariant determinants and factors of A(xo, x), re- 
spectively, fork = 1,... ,rank A(x, x,). They are called homogeneous determinants and the invariant 
homogeneous polynomials (factors), respectively, of A(xo, x). (Sometimes, 54 (x0, x1), 1¢(X0,X1),k = 
1,..., rank A(x, x;) are called the homogeneous determinants and the invariant homogeneous polyno- 
mials A(x).) 

Let A(x) € F[x]'*" and consider the module M C F [x]" ofall solutions of A(x)w(x) = 0. The set of 


all solutions w(x) is an F [x]-module M with a finite basis w;(x),... ,w;(x), where s = n — rank A(x). 
Choose a basis wi (x),... ,Ws(x) in M such that w;(x) € M has the lowest degree among all w(x) € M, 
which are linearly independent over F [x] of w1,... , Wr_1(x) fork = 1,... ,s. Then the column indices 
a) <a. <... <a, of A(x) are given as a, = deg wi(x),k = 1,...,5. The rowindices 0 < f; < B) < 


. < B; t =m—rank A(x), of A(x), are the column indices of A(x)". 


Facts: 


The notion of strict equivalence of n x n regular pencils over the fields goes back to K. Weierstrass [Wei67]. 
The notion of strict similarity of m x n matrices over the fields is due to L. Kronecker [Kro90]. Most of the 
details can be found in [Gan59]. Some special results are proven in [Frixx]. For information about matrix 
pencils over fields see Section 43.1. 


1. Let Ag, Ay, Bo, By € D”*", Then Ap + xA;~Bo + XBo <> Xp Ao + x1 A1~ Box + Bix). 

2. Let Ao, A; € D,. Then the invariant determinants and the invariant polynomials 65;(x9, x1), 
ig(XoX1), k = 1,...,rank x9 Ao + x) Aj, of x9 Ap + x; A; are homogeneous polynomials. More- 
over, if 54(x) and i,(x) are the invariant determinants and factors of the pencil Ap + x A, for 
k=1,...,rank Ap+x Aj, then 6;(x) = 6;(1,x), ix(x) = ix(1, x), fork = 1,...,rank Ap+x A). 

3. [Wei67] Let Ap +x A, € F [x]"*” bea regular pencil. Then a pencil By + xB, € F [x]"*" is strictly 
equivalent to Ag + x A; ifand only if Ap + x A; and By + x B, have the same invariant polynomials 
over F [x]. 

4. [Frixx] Let Aj +x A, By + xB, € D[x]"*". Assume that A,, B, € GL, (ID). Then Ap + xA1~ Bo + 
xB) => Ap txA, ~ Bo t+ xB. 

5. [Gan59] The column (row) indices are independent of a particular allowed choice of a basis 
wi (x),... Ws (x). 

6. For singular pencils the invariant homogeneous polynomials alone do not determine the class of 
strictly equivalent pencils. 

7. [Kro90], [Gan59] The pencils A(x), B(x) € F[x]’*" are strictly equivalent if and only if they have 
the same invariant homogeneous polynomials and the same row and column indices. 
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This chapter uses the notations, definitions, and facts given in Chapter 23. The aim of this chapter is to 
acquaint the reader with two difficult problems in matrix theory: 


1. Similarity of matrices over integral domains, which are not fields. 
2. Simultaneous similarity of tuples of matrices over C. 


Problem 1 is notoriously difficult. We show that for the local ring Ho this problem reduces to a Problem 
2 for certain kind of matrices. We then discuss certain special cases of Problem 2 as simultaneous similarity 
of tuples of matrices to upper triangular and diagonal matrices. The L -property of pairs of matrices, which 
is discussed next, is closely related to simultaneous similarity of pair of matrices to a diagonal pair. The 
rest of the chapter is devoted to a “solution” of the Problem 2, by the author, in terms of basic notions of 
algebraic geometry. 


24.1 Similarity of Matrices 


The classical result of K. Weierstrass [Wei67] states that the similarity class of A € F”*" is determined by 
the invariant factors of —A + xI, over F [x]. (See Chapter 6 and Chapter 23.) For a given A, B € F"*", 
one can easily determine if A and B are similar, by considering only the ranks of three specific matrices 
associated with A, B [GB77]. It is well known that it is a difficult problem to determine if A, B € D"*" are 
DD-similar for most integral domains that are not fields. The emphasis of this chapter is the similarity over 
the local field Ho. The subject of similarity of matrices over Hp arises naturally in theory linear differential 
equations having singularity with respect to a parameter. It was studied by Wasow in [Was63], [Was77], 
and [Was78]. 


Definitions: 


For E € D’”*",G € D?*4 extend the definition of the tensor or Kronecker product E @ G € D'?*"4 of 
E with G to the domain D in the obvious way. (See Section 10.4.) 
A, B € D™*" are called similar, denoted by A ~ B if B = QAQ7!, for some Q € GL,,(D). 
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Let A, B € H(Q)"*". Then A and B are called analytically similar, denoted as ANB, if A and B are 
similar over H(Q). 

A and B are called locally similar if for any ¢ € Q, the restrictions A;, B; of A, B to the local rings H;, 
respectively, are similar over H,. 

A, B are called point-wise similar if A(¢), B(¢) are similar matrices in C"*” for each ¢ € Q. 

A, B are called rationally similar, denoted as ANB, if A, B are similar over the quotient field M(Q) of 
H(Q). 

Let A, B € H?™*": 


AG) = S- Apr, Ix) = RA). “BRS” Bex*, - [l= RE): 
k=0 k=0 


Then n(A, B) and (A, B) are the index and the number of local invariant polynomials of degree p 
of the matrix I,, @ A(x) — B(x)’ @ Ih, respectively, for p = 0,1,.... 

(x) is called an algebraic function if there exists a monic polynomial p(A, x) = A"+7%_) di (x)A"-# € 
(C[x])[A] of A-degree n > 1 such that p(A(x), x) = 0 identically. Then A(x) is a multivalued function 
on C, which has n branches. At each point ¢ € C each branch Aj(x) of A(x) has Puiseaux expansion: 
Aj(x) = wear bji(S)(x — t)m, which converges for |x — ¢| < R(¢), and some integer m depending 
on p(x). \oi29 bjilS)(x — t)in is called the linear part of A; (x) at ¢. Two distinct branches 4 ;(x) and 
Ax(x) are called tangent at ¢ € C if the linear parts of 4.;(x) and Ax(x) coincide at ¢. Each branch A ;(x) 
has Puiseaux expansion around oo: 4 j(x) = x! Say cjix7™, which converges for |x| > R. Here, | is 
the smallest nonnegative integer such that c jg 4 0 at least for some branch Aj. eee jim is called 
the principal part of A ;(x) at oo. Two distinct branches A; (x) and Ax(x) are called tangent at oo if the 
principal parts of dj (x) and Ax (x) coincide at oo. 


Facts: 


The standard results on the tensor products can be found in Chapter 10 or Chapter 13 or in [MM64]. 
Most of the results of this section related to the analytic similarity over Ho are taken from [Fri80]. 


1. The similarity relation is an equivalence relation on D”™*”. 

2,A%B <> A(x) =—-A4+xI*B(x) = —B+<xI. 

3. Let A, B € F"*". Then A and B are similar if and only if the pencils —A + xI and —B + xI have 
the same invariant polynomials over F [x]. 

4, If E = [ej] €¢ D”*", G € D?*4, then E ® G can be viewed as the m x n block matrix [ei;G] pany. 
Alternatively, E ® G can be identified with the linear transformation 


L(E,G):D1*" > DPX", xX > GXE!?. 


5. (E @ G)(U @ V) = EU ® GV whenever the products EU and GV are defined. Also (E @ G)! 
= E' @G' (cf. §2.5.4)). 
6. For A, B € D"™", if A is similar to B, then 


I,@A-A'@Ql,~1,@A—B' Ql, ~1,®@B—B'@ Ih. 


7. [Gur80] There are examples over Euclidean domains for which the reverse of the implication in 
Fact 6 does not hold. 

8. [GB77] For D = F, the reverse of the implication in Fact 6 holds. 

9. [Fri80] Let Ae F”*”, B € F"*", Then 


1 
null (1, ® A— B! @ Im) < 5 (null (Im @ A— A’ @ Im) + null (1, @ B — B™ @I,)). 


Equality holds if and only if m = n and A and B are similar. 


Similarity of Families of Matrices 24-3 


10. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


Let A € F"*" and assume that p;(x),..., px(x) € F[x] are the nontrivial normalized invariant 
polynomials of —A+xJ, where pi(x)|p2(x)|...|pe(x) and pi(x) po(x)... pe(x) = det (xI— A). 
Then A © C(pi) @C(p2) ®... BC( px) and C( pi) ® C(p2) ®... B C(px) is called the rational 
canonical form of A (cf. Chapter 6.6). 

For A, B € H(Q)"*", analytic similarity implies local similarity, local similarity implies point-wise 
similarity, and point-wise similarity implies rational similarity. 

For n = 1, all the four concepts in Fact 11 are equivalent. For n > 2, local similarity, point-wise 
similarity, and rational similarity, are distinct (see Example 2). 

The equivalence of the three matrices in Fact 6 over H(Q) implies the point-wise similarity of A 
and B. 

Let A, B € Hj*”. Then A and B are analytically similar over Hp if and only if A and B are rationally 
similar over Ho and there exists 7(A, A) + 1 matrices To, ..., T;, € C"*" (1 = n(A, A)), such that 
det To 4 0 and 


k 
SO ATi Ty; B; = 0, k=0, ...,9(A, A). 
i=0 


Suppose that the characteristic polynomial of A(x) splits over Ho: 


det (AI — A(x)) = [[(A-Ai(x)),  ai(x) € Ho, i= 1,...,0. 


i=1 


Then A(x) is analytically similar to 


C(x) = BiCi(x), Ci(x) € Hy, 
(ajI,, — C;(0))” = 0, aj = Ay, (0), aj FO; for i # j, (he Rs Warman op 


Assume that the characteristic polynomial of A(x) € Hp splits in Hp.Then A(x) is analytically 
similar to a block diagonal matrix C(x) of the form Fact 15 such that each C;(x) is an upper 
triangular matrix whose off-diagonal entries are polynomials in x. Moreover, the degree of each 
polynomial entry above the diagonal in the matrix C; (x) does not exceed n(C;, C;) fori = 1, ...,€. 
Let P(x) and Q(x) be matrices of the form 


P(x) = OL Px), Pe) eH, 
(aj Im, — P;(0))" =0, a; F Qj for iF Pij=l,...sp, 


nj Xnj 
> 


Q(x) = O41, Qj(x), Q(x) € HI 
(Bj In, — Q;(0))" =0, B; F Bj for 1], Faw een G 


Assume furthermore that 
OQ; = Bi, 1 = Ty ests Qj # Bj, i=t+l, see Ds j=tt+l, ..5q9,0< t < min(p,q). 


Then the nonconstant local invariant polynomials of I @ P(x) — Q(x)’ @ I are the nonconstant 
local invariant polynomials of I @ P;(x) — Q;(x)’ @ I fori =1,...,t: 


KPO Sh) Pa Td 
i=1 


In particular, if C(x) is of the form in Fact 15, then 


n(C,C) = max (Cj, Ci). 
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18. A(x)© B(x) => Aly) ©B(y") for any2<meN. 

19. [GR65] (Weierstrass preparation theorem) For any monic polynomial p(A, x) = A"+3>}_, aj(x)A"! € 
Ho[A] there exists m € N such that p(A, y) splits over Ho. 

20. For a given rational canonical form A(x) € Hj** there are at most a countable number of analytic 
similarity classes. (See Example 3.) 

21. Fora given rational canonical form A(x) € Hj*”, where n > 3, there may exist a family of distinct 
similarity classes corresponding to a finite dimensional variety. (See Example 4.) 

22. Let A(x) € Hj*" and assume that the characteristic polynomial of A(x) splits in Ho as in Fact 15. 
Let B(x) = diag (Ai (x), ...,4n(x)). Then A(x) and B(x) are not analytically similar if and only 
if there exists a nonnegative integer p such that 


Kj(A, A) + Kj(B, B) = 2K,(A,B), 7 =0,...,p—1, ifp>1. 


In particular, A(x)©B (x) if and only if the three matrices given in Fact 6 are equivalent over Ho. 

23. [Fri78] Let A(x) € C[x]”*”. Then each eigenvalue A(x) of A(x) is an algebraic function. Assume 
that A(¢) is diagonalizable for some ¢ € C. Then the linear part of each branch of A; (x) is linear 
at C,i.e., is of the form a + Bx for some a, B € C. 

24. Let A(x) € C[x]"*” be of the form A(x) = sean Axx*, where Ay € C"™" fork = 0,...,£and 
£> 1, Ae £ 0. Then one of the following conditions imply that A(x) = S(x)B(x)S7~!(x), where 
S(x) € GL(n, C[x]) and B(x) € C[x]"*" is a diagonal matrix of the form @71,A;(x)J;,, where 
ky,... km > 1. Furthermore, A(x), ...,Am(x) are m distinct polynomials satisfying the following 
conditions: 


(a) deg A; = € > deg Aj(x),i = 2,...,m—1. 
(b) The polynomial Aj (x) —A (x) has only simple roots in C fori # j.(Ai(f) =Aj(E) SAO) F 
ng). 

i. The characteristic polynomial of A(x) splits in C[x], i-e., all the eigenvalues of A(x) are 
polynomials. A(x) is point-wise diagonalizable in C and no two distinct eigenvalues are 
tangent atany¢ €C. 

ii. A(x) is point-wise diagonalizable in C and Ay is diagonalizable. No two distinct eigenvalues 


are tangent at any point ¢ € C U {oo}. Then A(x) is strictly similar to B(x), i.e., S(x) can 
be chosen in GL(n, C). Furthermore, A) (x), ...;Am(x) satisfy the additional condition: 


. 5 Pxe e dé e dé e—1 do 
(c) Fori # j, either 4* (0) 4 Si (0) or S#(0) = Fi(0) and 4=* (0) 4 S(O). 
Examples: 
1. Let 


A= 


1 0 eal ar eee es 
0 5] 05 


Then A(x) and B(x) have the same invariant polynomials over Z[x] and A and B are not similar 


over Z. 
01 z 0 
» D(z= : 
0 0 0 1 


2. Let 
Then zA(z) = D(z)A(z)D(z)!, ite., A(z), zA(z) are rationally similar. Clearly A(z) and zA(z) 
are not point-wise similar for any Q containing 0. Now zA(z), 2 A(z) are point-wise similar in C, 
but they are not locally similar on Ho. 


A(z) = 
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3. Let A(x) € H3*? and assume that det (AI — A(x)) = (A — Aj(x))(A — A2(x)). Then A(x) is 
analytically similar either to a diagonal matrix or to 


sagas k=0 (p > 0) 
AS" eae pans PEO): 


Furthermore, if A(x)&B(x), then n(A, A) =k. 
4. Let A(x) € H*?. Assume that 


A(x)¥C(p),  p(A,x) =A(A—?™)(Q—x™), mE 1. 
Then A(x) is analytically similar to a matrix 


0 x a(x) 
B(x,a)= ]0 x2" x2 1) 0 <k,k < 00 (x® =0), 


0 Ox 


where a(x) is a polynomial of degree 4m — 1 at most. Furthermore, B(x,a )XB (x, b) ifand only if 
(a) Ifa(0) 4 1, then b — a is divisible by x”. 


(b) Ifa(0) =1and $4 =0, i=l, vk —1, £4 £0 for 1 <k <m, then b — ais divisible by x”"**. 


(c) Ifa(0) = 1 and oe =0,i=1,...,m, then b — a is divisible by x””. 


Then for k; = ky = mand a(0) € C\{1}, we can assume that a(x) is a polynomial of degree less 
than m. Furthermore, the similarity classes of A(x) are uniquely determined by such a(x). These 
similarity classes are parameterized by C\{1} x C7! (the Taylor coefficients of a(x)). 


24.2 Simultaneous Similarity of Matrices 


In this section, we introduce the notion of simultaneous similarity of matrices over a domain D. The 
problem of simultaneous similarity of matrices over a field F, i.e., to describe the similarity class of a 
given m (> 2) tuple of matrices or to decide when a given two tuples of matrices are simultaneously 
similar, is in general a hard problem, which will be discussed in the next sections. There are some cases 
where this problem has a relatively simple solution. As shown below, the problem of analytic similarity of 
A(x), B(x) € H5*" reduces to the problem of simultaneously similarity of certain 2-tuples of matrices. 


Definitions: 


For Ao, ..., Ay € D"*" denote by A(Apo, ..., Ar) C D"*" the minimal algebra in D"*” containing J, and 
Ao, ..., Ay. Thus, every matrix G € A(Apo, ..., Ay) isa noncommutative polynomial in Ao, ..., Aj. 

For 1 > 1, (Ao, Ai, ---> Ag), (Bo, --., Be) € (D"*")**! are called simultaneously similar, denoted by 
(Ao, Ai, -.-, Ag) © (Bo, ..., Be), if there exists P € GL(n,D) such that B; = PA;P~!,i =0,...,€,i.e., 
(Bo, By, ..., Be) = P(Ao, Al, ..-, Ag) Po. 

Associate with (Ag, Aj, ..., Ac), (Bo, ..-» Be) € (D”*")‘*! the matrix polynomials A(x) = oy A;x', 
B(x) = ae B;x' € D[x]"*". A(x) and B(x) are called strictly similar (AXB) if there exists P € 
GL(n, D) such that B(x) = P A(x)P7!. 


Facts: 
1, AB <=> (Ao, Ai, .-.5 Ae) © (Bos «+> Be) 
2. (Ao, ...> Ae) € (C"*")**? is simultaneously similar to a diagonal tuple (Bo, ..., Be) € (C”*")*1, 
ie., each B; is a diagonal matrix if and only if Ao, ..., Ag are € + 1 commuting diagonalizable 
matrices: Aj A; = Aj Aj fori, j =0,...,2. 
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3. If Ao, ..., Ae € C”*" commute, then (Ao, ..., Ag) is simultaneously similar to an upper triangular 
tuple (Bo, io Bh0 5 By). 

4. Let! € N, (Ao)... , Ar), (Bo,... By) € (C™), and U = [Uj ]ISLV = [Vilitl, W = 
[Wijlitts € CMD xnd+) | Uij, Vij, Wij € C"*™",i,j = 1,...,1 + 1 be block upper triangular 

matrices with the following block entries: 


Ui; = Aj-is V;; = B;-i, Wij = 6G4njIn i= | een J+lyj=i... e+. 


Then the system in Fact 14 of section 24.1 is solvable with Ty € GL(n, C) ifand only for! = «(A, A) 
the pairs (U, W) and (V, W) are simultaneously similar. 
5. For Ag, ..., Ae € (C"*")'*! TFAE: 


* (Ao, ..., Ae) is simultaneously similar to an upper triangular tuple (Bo, ..., Be) € (C™")*!. 


° Forany0 <i <j <€and Me A(Ap, ..., Ag), the matrix 
(A; Aj — Aj Aj)M is nilpotent. 


6. Let X) = A(Ao, ..., Ag) © F"*" and define recursively 
A= So (AA) — AjAD M1 SF", k=1,.... 
O0<i<j<é 


Then (Ao, ..., Ag) is simultaneously similar to an upper triangular tuple ifand only if the following 
two conditions hold: 


© AX CM, i=0,...,€,k=0,.... 
* There exists q > 1 such that XY, = {0} and A; is a strict subspace of 4; fork = 1, ...,q. 


Examples: 
: : ; : : 1 2 
1. This example illustrates the construction of the matrices U and W in Fact 4. Let Ap = , {| ; 
5 6 1 -l 
Aj = 7 gan a= 1 7 
12 5 6 1 -l 0 0 1 0 0 0 
3 4 7 8 -1 1 0 00 1 0 0 
0 0 1 2 5 6 0 00 0 1 =0 
WO gcd sg sph Se oe tg prt 
0 0 0 0 1 2 0 00 0 0 0 
0 0 0 0 3 4 0 00 0 0 0 


24.3 Property L 


Property L was introduced and studied in [MT52] and [MT55]. In this section, we consider only square 
pencils A(x) = Ap + Ayx € C[x]"*", A(x9, x1) € C[xo, x1 ]"*", where A; 4 0. 


Definitions: 


A pencil A(x) € C[x]"*” has property L if all the eigenvalues of A(xo, x;) are linear functions. That is, 
Ai(xo, X1) = aX + B;x) is an eigenvalue of A(xo, x,) of multiplicity n; fori = 1, ...,m, where 


n= >on (a, Bi) A (aj, Bj), for 1<i <j<m. 


i=1 


A pencil A(x) = Ay + A,x is Hermitian if A°, A! are Hermitian. 
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Facts: 


Most of the results of this section can be found in [MF80], [Fri81], and [Frixx]. 


1. 


For a pencil A(x) = Ap + x A, € C[x]"*" TFAE: 

° A(x) has property L. 

* The eigenvalues of A(x) are polynomials of degree 1 at most. 

* The characteristic polynomial of A(x) splits into linear factors over C[x]. 

* There is an ordering of the eigenvalues of Ap and Aj, a, ...,a@, and fj, ...,B,, respectively, 
such that the eigenvalues of Agxo + Ajx; are @)X9 + Bix], ...,QnXo + Bux). 


A pencil in A(x) has property L if one of the following conditions hold: 


¢ A(x) is similar over C(x) to an upper triangular matrix U(x) € C(x)"*". 
* A(x) is strictly similar to an upper triangular pencil U(x) = Up + Uix. 
nxn 


° A(x) is similar over C[x] to a diagonal matrix B(x) € C[x] 


* A(x) is strictly similar to diagonal pencil. 


. Ifa pencil A(xo, x;) has property L, then any two distinct eigenvalues are not tangent at any point 


of CU. 


. Assume that A(x) is point-wise diagonalizable on C. Then A(x) has property L. Furthermore, 


A(x) is similar over C[x] to a diagonal pencil B(x) = By + B,x. Suppose furthermore that A, is 
diagonalizable, i.e., A(xo, x1) is point-wise diagonalizable on C?. Then A(x) is strictly similar to a 
diagonal pencil B(x), i.e, Ag and A; are commuting diagonalizable matrices. 


5. Let A(x) = Ap + Aix € C[x]"*" such that A; and A: are diagonalizable and Ap A; # A; Ao. Then 

exactly one of the following conditions hold: 

* A(x) is not diagonalizable exactly at the points ¢, ...,¢), where 1 < p < n(n — 1). 

* Forn = 3, A(x) € C[x]"*” is diagonalizable exactly at the points ¢; = 0, ...,, forsomeq > 1. 
(We do not know if this condition is satisfied for some pencil.) 

6. Let A(x) = Ap + A;x bea Hermitian pencil satisfying Ag A; # A; Ao. Then there exists 2q distinct 
complex points ¢),¢,..., Sask e€C\R l1<q< mod) such that A(x) is not diagonalizable if 
and only if x € {f,¢),.. bqrSqh- 

Examples: 
1. This example illustrates the case n = 2 of Fact 5. Let 


1 
3 


1 3 
—3 1 


x+1 3x 


A — 
: —3x x+2 


2 
and A,; = 


| » so A(x) = 


Foré € C, the only possible way A(¢) can fail to be diagonalizable is if A(¢ ) has repeated eigenvalues. 


The eigenvalues of A(¢) are 5 (2¢ — /f/1— 3662+ 3) and 5 (2¢ + /1— 3667+ 3) so the only 
values of ¢ at which it is possible that A(¢) is not diagonalizeable are ¢ = +i, and in fact A(+2) 
is not diagonalizable. 


24.4 Simultaneous Similarity Classification I 


This section outlines the setting for the classification of conjugacy classes of 1+ 1 tuples (Ao, Ai,..., A) € 
(c"*")'*! under the simultaneous similarity. This classification depends on certain standard notions in 
algebraic geometry that are explained briefly in this section. A detailed solution to the classification of 
conjugacy classes of / + 1 tuples is outlined in the next section. 
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Definitions: 


X CCN is called an affine algebraic variety (called here a variety) if it is the zero set of a finite number 
of polynomial equations in CN. 

# is irreducible if Y does not decompose in a nontrivial way to a union of two varieties. 

If & is a finite nontrivial union of irreducible varieties, these irreducible varieties are called the irre- 
ducible components of 1. 

x € X is called a regular (smooth) point of irreducible ¥ if in the neighborhood of this point ¥ is a 
complex manifold of a fixed dimension d, which is called the dimension of ¥ and is denoted by dim 7. 
Y is an irreducible variety of dimension —1. 

For a reducible variety V C CN, the dimension of ), denoted by dim Y, is the maximum dimension 
of its irreducible components. 

The set of singular (nonsmooth) points of V is denoted by 4;. 

A set Z is a quasi-irreducible variety if there exists a nonempty irreducible variety V and a strict 
subvariety Y C ¥ such that Z = V\JY. The dimension of Z, denoted by dim Z, is defined to be equal to 
the dimension of ¥. 

A quasi-irreducible variety Z is regular if Z C V\A;. 

A stratification of C is a decomposition of C to a finite disjoint union of %,... ,X, of regular 
quasi-irreducible varieties such that Cl (4})\4 = Uje4,¥} for some A; C {1,..., p} fori = 1,..., p. 
(Cl (4) = 4 == Aj; = 9B.) 

Denote by C[C] the ring of polynomial in N variables with coefficients in C. 

Denote by W,,)+41,-+1 the finite dimensional vector space of multilinear polynomials in (J+ 1)n? variables 
of degree at most r + 1. That is, the degree of each variable in any polynomial is at most 1. N(n,1,r) := 
dim Wyp4ir+i- Waj+ir+1 has a standard basis e1,... , en(nj,r) in Wrj+iy+41 Consisting of monomials in 
(1 + 1)n? variables of degree r + 1 at most, arranged in a lexicographical order. 

Let ¥ C CX bea quasi-irreducible variety. Denote by C[4’] the restriction of all polynomials f(x) € 
C[C] to ¥, where f,g € C[CN] are identified if f — g vanishes on ¥. Let C() denote the quotient 
field of C[¥]. 

A rational function h € C(%X) is regular if h is defined everywhere in 1’. A regular rational function on 
& is an analytic function. 

Denote by A the / + 1 tuple (Ap,..., Ay) € (c"™")/+1_ The group GL(n, C) acts by conjugation on 
(c™")41: TAT"! = (TAT !,... TA, T—!) for any A € (C"™")'*! and T € GL(n, C). 

Let orb (A) := {TAT~!: T € GL(n, C)} be the orbit of A (under the action of GL(n, C)). 

Let XY c (C"*")'+! bea quasi-irreducible variety. ¥ is called invariant (under the action of GL(n, C)) 
if TXT~! = & forall T € GL(n, C). 

Assume that ¥ is an invariant quasi-irreducible variety. A rational function h € C(%) is called invariant 
if h is the same value on any two points of a given orbit in 1’, where h is defined. Denote by C[4’]}*" 
C[X’] and C(X)™ C C(X) the subdomain of invariant polynomials and subfield of invariant functions, 
respectively. 


Facts: 


For general background, consult for example [Sha77]. More specific details are given in [Fri83], [Fri85], 
and [Fri86]. 


. An intersection of a finite or infinite number of varieties is a variety, which can be an empty set. 

. A finite union of varieties in CY is a variety. 

. Every variety ¥ is a finite nontrivial union of irreducible varieties. 

Let Y c CN be an irreducible variety. Then ¥ is path-wise connected. 

&; is a proper subvariety of the variety VY and dim ¥, < dim ¥. 

. dim CN = Nand (CX), = &. For any Z € CN, the set {z} is an irreducible variety of dimension 0. 
. A quasi-irreducible variety Z = %\Y) is path-wise connected and its closure, denoted by Cl (Z), 
is equal to V’. Cl (Z)\Z is a variety of dimension strictly less than the dimension of Z. 


NDUBwWN EE 
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The set of all regular points of an irreducible variety Y, denoted by ¥, := V\A;,, is a quasi- 
irreducible variety. Moreover, 1, is a path-wise connected complex manifold of complex dimension 
dim ¥. 

N(abr) = dim Wasi = td (AP), 


1 


. For an irreducible ¥, C[’] is an integral domain. 

. For a quasi-irreducible V, C[4’],C(¥) can be identified with C[Cl (4)], C(Cl (4’)), respectively. 
. For A € (C”™")/+1, orb (A) is a quasi-irreducible variety in (C”*")'*!. 

. Let X¥ c (C"*")!t! bea quasi-irreducible variety. V is invariant ifA € 4% <=> orb (A) CX. 

. Let ¥ bean invariant quasi-irreducible variety. The quotient field of C[¥’] inv is a subfield of C(XV)'™, 


and in some interesting cases the quotient field of C[.Y ]” is a strict subfield of C(¥V)i™. 


. Assume that ¥ Cc (C"*")!+! is an invariant quasi-irreducible variety. Then C[4’]" and C(4’)" 


are finitely invariant generated. That is, there exists f\,..., f; € C[¥V }’ and Zire Bj ECL yinv 
such that any polynomial in C[4’]™ is a polynomial in fi,..., f;, and any rational function in 
C(X)*’ is a rational function in Zire je 


16. (Classification Theorem) Let n > 2 and/ > 0 be fixed integers. Then there exists a stratification 
UP aX; of (C"*")!+! with the following properties. For each ¥; there exist m; regular rational 
functions g1j,---52m,i € C(X;)” such that the values of gji for j = 1,... ,m; on any orbit in 
%; determines this orbit uniquely. 

The rational functions g1j,... ; m,,i are the generators of C(A;)" fori =1,..., p. 
Examples: 
1. Let S bean irreducible variety of scalar matrices S := {A € C?*?: A= tb} and X := C?%*\S 


be a quasi-irreducible variety. Then dim ¥ = 4, dim S = 1, and C?*? = X US isa stratification 
of C2*?, 


. Let c (C2?) be the set of all pairs (A, B) € (C?*?)?, which are simultaneously similar to a pair 


of upper triangular matrices. Then U/ is a variety given by the zero of the following polynomial: 
U := {(A, B) € (C?**)? : (2tr A? — (tr A)*)(2tr B? — (tr B)?)—(2tr AB—tr Atr B)* = 0}. 
LetC CU be the variety of commuting matrices: 
C := {(A,B) € (C’?’)? : AB— BA=0}. 
Let V be the variety given by the zeros of the following three polynomials: 
V := {(A, B) € (C?*?)? : 2tr A? — (tr A)? = 2tr B*— (tr BY? =2tr AB—tr Atr B = 0}. 
Then V is the variety of all pairs (A, B), which are simultaneously similar to a pair of the form 


re 


OA 


S Cc W be defined as in the previous example. Define the following quasi-irreducible varieties in 
(C2*?)?2: 


| : |; al Hence, V C C. Let W := {A € (C2*%*) : 2tr A? — (tr A)? = 0} and 


xX — (C2%?)"\U, xX = Uu\Cc, Xx; = C\Y, X4 = V\(S x WUW x S), 
Xs i= S X (W\S), Xe = (W\S) x S,47=SxS. 


Then 


dim * = 8, dim %) = 7, dim 13 = 6, dim 44 = 5, 
dim Xs = dim X = 4, dim xX = 2 


and U?_, 4; isa stratification of (C?*?)?. 


. In the classical case of similarity classes in C"*", i.e., 1 = 0, it is possible to choose a fixed set of 


polynomial invariant functions as g;(A) = tr (A!) for j = 1,...,n. However, we still have to 
stratify C"*" to UP vs where each A € %; has some specific Jordan structures. 
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4. Consider the stratification C?*? = ¥ U S as in Example 1. Clearly ¥ and S are invariant under 
the action of GL(2, C). The invariant functions tr A,tr A* determine uniquely orb (A) on V. The 
Jordan canonical for of any A in ¥ is either consists of two distinct Jordan blocks of order 1 or 
one Jordan block of order 2. The invariant function tr A determines orb (A) for any A € S. It is 
possible to refine the stratification of C?*? to three invariant components C?*?\W, W\S, S, where 
W is defined in Example 2. Each component contains only matrices with one kind of Jordan block. 
On the first component, tr A, tr A?’ determine the orbit, and on the second and third component, 
tr A determines the orbit. 

5. To see the fundamental difference between similarity (1 = 0) and simultaneous similarity / > 1, 
it is suffice to consider Example 2. Observe first that the stratification of (C?*?)? = U7_, 4; is 
invariant under the action of GL(2,C). On 4 the five invariant polynomials tr A, tr A’, tr B, 
tr B*,tr AB, which are algebraically independent, determine uniquely any orbit in 4}. 

Let (A = [ajj],B = [bjj]) € 4%. Then A and B have a unique one-dimensional common 
eigenspace corresponding to the eigenvalues Aj, 1; of A, B, respectively. Assume that aj.b2) — 
4b, # 0. Define 


by, — ba7)a12@21 + €22412b2) — ay, a2\b 
dy =a(A, B):= (bir = Bop )ainaar + az2a12ba1 — ai1a21 2 
@12b21 — ay1b 12 


al =a(B, A). 


Thentr A, tr B, a(A, B), a(B, A) are regular, algebraically independent, rational invariant func- 
tions on 1X), whose values determine orb (A, B). Cl (orb (A, B)) contains an orbit generated by two 
diagonal matrices diag (Aj, 42) and diag (11, 42). Hence, C[X5]'™’ is generated by the five invariant 
polynomials tr A, tr A’,tr B,tr B?,tr AB,whichare algebraically dependent. Their values coin- 
cide exactly on two distinct orbits in 4. On 43 the above invariant polynomials separate the orbits. 


Any (A= [a;;], B = [b;;]) € ¥1issimultaneously similar a unique pair of the form (F A ; |, i) . 


Thent = y(A, B) := ba Thus, tr A,tr B,y(A, B) are three algebraically independent regular ra- 
tional invariant functions on 14, whose values determine a unique orbit in 14. Clearly (Ah, wl) € 
Cl (X,). Then C[A,]”” is generated bytr A,tr B.Thevaluesoftr A = 2A,tr B = 2 correspond 
toa complex line of orbits in 4’,. Hence, the classification problem of simultaneous similarity classes 
in %, or V is a wild problem. 

On %5, X, 7, the algebraically independent functions tr A,tr B determine the orbit in each 
of the stratum. 


24.5 Simultaneous Similarity Classification II 


In this section, we give an invariant stratification of (C"*")I+1, for ] > 1, under the action of GL(n, C) 
and describe a set of invariant regular rational functions on each stratum, which separate the orbits up to 
a finite number. We assume the nontrivial case n > 1. It is conjectured that the continuous invariants of 
the given orbit determine uniquely the orbit on each stratum given in the Classification Theorem. 

Classification of simultaneous similarity classes of matrices is a known wild problem [GP69]. For 
another approach to classification of simultaneous similarity classes of matrices using Belitskii reduction 
see [Ser00]. See other applications of these techniques to classifications of linear systems [Fri85] and to 
canonical forms [Fri86]. 


Definitions: 

For A = (Ao,...,Az),B = (Bo... , By) € (C™")+! let L(B, A) : C"*" > (C"")!+! be the linear 
operator given by U + (By U —UAg,... , B}U — UA)). Then L (B, A) is represented by the (J ++ 1)n? x n? 
matrix (In @ BU — Ap ® Ins... » In @ B} — Ay @ In)’, where U +> (In ® Bd — Ap @ Iny--- 5 In @ BE — 
A; ® I,)™U. Let L(A) := L(A, A). The dimension of orb (A) is denoted by dim orb (A). 
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LetS, = {AeEC™": A= ©4T,} be the variety of scalar matrices. Let 


Marri = {A € (C™")4! : rank L(A) =r}, 1 =0,1,...,n7-1. 


Facts: 
Most of the results in this section are given in [Fri83]. 


1. For A= (Ap,... , Ay), € (C"*")/*!, dim orb (A) is equal to the rank of L(A). 
2. SinceanyU € S, commutes withany B € C””” it follows that ker L(A) D S,,. Hence, rank L(A) < 


n—1 
3. Maj+i2—1 1s a invariant quasi-irreducible variety of dimension (J + 1)n?, ie, Cl (Mapsie—1) = 
(c™")41, The sets Majsip? = 1? — 2,...,0 have the decomposition to invariant quasi- 


irreducible varieties, each of dimension strictly less than (J + 1)n’. 

4, Letr € [0,n?—1],A € Myys1,, and B = TAT—!. Then L(B,A) = diag (I, @ T,... I, ® 
T)L(A)U, @ T7'), rank L(B,A) = r and det L(B, A)[o, 8] = 0 for anya € Q4i4i)n2B € 
Q,+41,n2- (See Chapter 23.2) 

5. Let X = (Xo,... , Xy) € (C"*")!+! with the indeterminate entries X,; = [x,,ij] fork = 0,... ,1. 
Each det L(X, A)[a, BJ,@ € Qr4i(i4i)n2s B € Q-41n2 isa vector in Wy741,41) Le., it is a multilinear 
polynomial in (J + 1)n* variables of degree r + 1 at most. We identify det L(X,A)[a, B],a € 
Q-41,041)r2>B € Qr+41,n2 With the row vector a(A,a@, 8) € CMabr) given by its coefficients in the 
basis e1,... 5 €n(nJ,r). The number of these vectors is M(n,l,r) := (eee) (a) Let R(A) € 
CM@L NGL) xMab NL) be the matrix with the rows a(A,a, 8), where the pairs (a,8) € 
Qr+i+ rn X Qr4iy2 are listed in a lexicographical order. 

6. All points on the orb (A) satisfy the following polynomial equations in ci(c**")4 1}: 


det L(X,A)[a, 6] = 0, 
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Thus, the matrix R(A) determines the above variety. 

7. IfB = TAT, then R(A) is row equivalent to R(B). To each orb (A) we can associate a unique 
reduced row echelon form F (A) € CM") N(@uhr)xM(@bLr Nr) of R(A). 9(A) := rank R(A) is the 
number oflinearly independent polynomials given in (24.1). LetZ(A) = {(1, 1),.-- »(@(A), jocay)} C 
{1,...,Q@(A)} x {1,..., N(,1,1r)} be the location of the pivots in the M(n,1,r) x N(n,1,r) matrix 
F(A) = [fij(A)]. That is, 1 < j) < ... < joa) < Nn, lr), fij(A) = 1 fori = 1,...,e(A) 
and fj; = 0 unless j > i andi € [1,@(A)]. The nontrivial entries fj;(A) for j > i are rational 
functions in the entries of the / + 1 tuple A. Thus, F(B) = F(A) for B € orb (A). The numbers 
r(A) := rank L(A), 9(A) and the set Z(A) are called the discrete invariants of orb (A). The rational 
functions fjj(A),i = 1,...,e(A),j =it+1,...,N(n,1,r) are called the continuous invariants 
of orb (A). 

8. (Classification Theorem for Simultaneous Similarity) Let 1 > 1,n > 2 be integers. Fix an in- 
teger r € [0, n? — 1] and let M(n,1,r), N(n,l,r) be the integers defined as above. Let 0 < 
@ < min(M(n,l,r), N(n,1,r)) and the set J = {(1,j)),--.5(@ jo) C {1,-..,0} x {1..., 
N(n hr), < fi < ... < jo < N(n,l,r) be given. Let MyJ41,(@,Z) be the set of all 
A € (C"*")!+! such that rank L(A) = r, @(A) = 9, and Z(A) = Z. Then M,141,(0,Z) is 
invariant quasi-irreducible variety under the action of GL(n, C). Suppose that Mj141.,(0,Z) 4 Y. 
Recall that for each A € M,,141.,(Q,Z) the continuous invariants of A, which correspond to the en- 
tries fij(A), i=1,...,0, j =it+1,...,N(n,1,r) of the reduced row echelon form of R(A), are 
regular rational invariant functions on M,,41,, (0, Z). Then the values of the continuous invariants 
determine a finite number of orbits in. M,141,-(0,Z). 

The quasi-irreducible variety M,,741,(@,Z) decomposes uniquely as a finite union of invariant 
regular quasi-irreducible varieties. The union of all these decompositions of M41, (0,Z) for all 
possible values r, 9, and the sets Z gives rise to an invariant stratification of (Cox yi+1, 
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Max-plus algebra has been discovered more or less independently by several schools, in relation with various 
mathematical fields. This chapter is limited to finite dimensional linear algebra. For more information, the 
reader may consult the books [CG79], [Zim81], [CKR84], [BCOQ92], [KM97], [GM02], and [HOvdW06]. 
The collections of articles [MS92], [Gun98], and [LM05] give a good idea of current developments. 


25.1 Preliminaries 


Definitions: 


The max-plus semiring Ryyx is the set R U {—o0o}, equipped with the addition (a,b) t+ max(a,b) and 
the multiplication (a,b) > a+ b. The identity element for the addition, zero, is —0o, and the identity 
element for the multiplication, unit, is 0. To illuminate the linear algebraic nature of the results, the generic 
notations ++, }), x (or concatenation), Oand Lare used for the addition, the sum, the multiplication, the 
zero, and the unit of Rnax, respectively, so that when a, b belong to Rmax, a + b will mean max(a, b),a « b 
or ab will mean the usual sum a + b. We use blackboard (double struck) fonts to denote the max-plus 
operations (compare “++” with “+”). 

The min-plus semiring Rmin is the set R U {-++-0o} equipped with the addition (a,b) t+ min(a,b) 
and the multiplication (a,b) % a + b. The zero is +00, the unit 0. The name tropical is now also used 
essentially as a synonym of min-plus. Properly speaking, it refers to the tropical semiring, which is the 
subsemiring of Rmin consisting of the elements in N U {+00}. 

The completed max-plus semiring R,,, is the set R U {--oo} equipped with the addition (a,b) 
max(a, b) andthe multiplication (a, b) & a+b, with the convention that —oo+ (+00) = +00+(—0o0) = 
—oo. The completed min-plus semiring, Rynin, is defined in a dual way. 

Many classical algebraic definitions have max-plus analogues. For instance, Ri, is the set of n- 


max 


dimensional vectors and R”*? is the set of n x p matrices with entries in Rmax. They are equipped 
with the vector and matrix operations, defined and denoted in the usual way. The n x p zero matrix, 0,5 


or 0, has all its entries equal to O. The n x n identity matrix, I, or I, has diagonal entries equal to 1, and 
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nondiagonal entries equal to O. Given a matrix A = (Ajj) € RiP, we denote by Aj. and A.; the i-th row 
and the j-th column of A. We also denote by A the lineae map R?,, > Ri, sending a vector x to Ax. 
Semimodules and subsemimodules over the semiring Ryax are defined as the analogues of modules and 
submodules over rings. A subset F of a semimodule M over Rmax spans M, or is a spanning family of M, 
if every element x of M can be expressed as a finite linear combination of the elements of F , meaning that 
x= Ne pr Ag-f, where (Ag)fer is a family of elements of Rmax such that Ag = O for all but finitely many 
f <¢ F. Asemimodule is finitely generated if it has a finite spanning family. 

The sets Rmax and Rmax are ordered by the usual order of R U {+00}. Vectors and matrices over Rinax are 
ordered with the product ordering. The supremum and the infimum operations are denoted by v and A, 
respectively. Moreover, the sum of the elements of an arbitrary set X of scalars, vectors, or matrices with 


entries in Ryax is by definition the supremum of X. 
Rr xn 


If A € R,,,,> the Kleene star of A is the matrix A* = I + A+ A? +- 

The digraph I'(A) associated to an n x n matrix A with entries in ae eeages of the vertices 1,... ,7, 
with an arc from vertex i to vertex j when A;; 4 O. The weight of a walk W given by (11, i2),... , (ik—1, ik) 
is|Wla := Aji, +++ Ai,_ji,» and its length is |W| := k — 1. The matrix A is irreducible if I'(A) is strongly 
connected. 

Facts: 
1. When A € Rr, the oe of a walk W = ((i),72),... 5 (4e-1,i%)) in '(A) is given by the usual 


sum |W|4 = Aji, +--+ + Ai,_,i,. and Pave gives the pea “cient |W], of a walk from vertex i 


to vertex j. One can also define the matrix A* when A € R”.”. Then, Aj, is the minimal weight of 
a walk from vertex i to vertex 7. Computing A” is the same as scthe all pains * shortest path problem. 

2. [CG79], [BCOQ92, Th. 3.20] If A € Rr and the weights of the cycles of (A) do not exceed 1, 
then A*= I++ A+---+ A™?, 

3. [BCOQ92, Th. 4.75 and Rk. 80] If A € Roe and be Rae then the smallest x € R ax such that 
x = Ax-+b coincides with the smallest x € R,,,, such that x > Ax-t b, and it is given by A*b. 

4. [BCOQ92, Th. 3.17] When A € Ri*’, be ae and when all the cycles of (A) have a weight 
strictly less than 1, then A*b is the unique solution x € Ri,,, of x = Ax-+b. 

5. Let A € R?*" and b € R’_,,. Construct the sequence: 


max" 


Xo = b, x} = Axo ++b, x» = Ax; +b,.... 


The sequence x; is nondecreasing. If all the cycles of (A) have a weight less than or equal to 1, 
then, X,-) = X, = ++: = A*b. Otherwise, x,_; 4 x,. Computing the sequence x; to determine 
A*b is a special instance of label correcting shortest path algorithm [GP88]. 

6. [BCOQ92, Lemma 4.101] For alla € R”*" be Rice Re" ,anddeé Re” "we have 


max? max ? 


a bl" fa*+a*b(cath+d)*ca* a*h(ca*h+d)* 
c dl (ca*b +d)*ca* (ca*b-+d)* 


This fact and the next one are special instances of well-known results of language theory [Eil74], 
concerning unambiguous rational identities. Both are valid in more general semirings. 
7. [MY60] Let A € TR Construct the sequence of matrices A, ..., A such that AM = A and 


AW =A: Dg Abe Dag DNs Ay 1) 
ij * 


fori,j =1,...,nandk=1,...,n.Then, A”) = A+ A?+4---. 
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Example: 
1. Consider the matrix 
4 3 
A= . 
7 4 
The digraph I’(A) is 
3 
a! 
4ca, 7 @ 
Me A ee 


We have 


a 10 7 
~ 41 104] 


For instance, Aj, = Ay. Ay = [4 3][4 7]? = max(4+ 4,347) = 10. This gives the maximal 
weight of a walk of length 2 from vertex 1 to vertex 1, which is attained by the walk (1, 2), (2, 1). 
Since there is one cycle with positive weight in (A) (for instance, the cycle (1, 1) has weight 4), 
and since A is irreducible, the matrix A* has all its entries equal to +-oo. To get a Kleene star with 
finite entries, consider the matrix 
—l -2 
2 oo] 


The only cycles in (A) are (1, 1) and (1,2), (2, 1) (up to a cyclic conjugacy). They have weights 
—1and 0. Applying Fact 2, we get 


Applications: 


1. Dynamic programming. Consider a deterministic Markov decision process with a set of states 
{1,... , a} in which one player can move from state to state j, receiving a payoff of Aj; € RU{—oo}. 
To every state 7, associate an initial payoff c; € RU {—oo} and a terminal payoff bj € RU {—oo}. 
The value in horizon k is by definition the maximum of the sums of the payoffs (including the 
initial and terminal payoffs) corresponding to all the trajectories consisting exactly of k moves. It is 
given by cA*b, where the product and the power are understood in the max-plus sense. The special 
case where the initial state is equal to some given m € {1,... ,n} (and where there is no initial 
payoff) can be modeled by taking c := e,,, the m-th max-plus basis vector (whose entries are all 
equal to O, except the m-th entry, which is equal to 1). The case where the final state is fixed can be 
represented in a dual way. Deterministic Markov decision problems (which are the same as short- 
est path problems) are ubiquitous in operations research, mathematical economics, and optimal 
control. 

2. [BCOQ92] Discrete event systems. Consider a system in which certain repetitive events, denoted 
by 1,...,1, occur. To every event i is associated a dater function x; : Z — R, where x;(k) rep- 
resents the date of the k-th occurrence of event i. Precedence constraints between the repetitive 
events are given by a set of arcs E C {l,..., n}*, equipped with two valuations v : E > N and 
t: E > R.If (i,j) € E, the k-th execution of event i cannot occur earlier than 7;; time units 
before the (k — v;;)-th execution of event j, so that x(k) => max;. (j,j)ez Tij + xj(k — vjj). This 
can be rewritten, using the max-plus notation, as 


x(k) > Aox(k) 44+ ++ +b Apx(k — 0), 
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is the vector with entries x;(k). Often, the dates 
x;(k) are only defined for positive k, then appropriate initial conditions must be incorporated 
in the model. One is particularly interested in the earliest dynamics, which, by Fact 3, is given by 
x(k) = Aj Aix(k—1) 4+- +--+ Aj Apx(k—0). The class of systems following dynamics of these forms 
is known in the Petri net literature as timed event graphs. It is used to model certain manufacturing 
systems [CDQV85], or transportation or communication networks [BCOQ92], [HOvdW06]. 


where ) := max(j,jyez vij and x(k) € Rix 


. N. Bacaér [Bac03] observed that max-plus algebra appears in a familiar problem, crop rotation. 


Suppose n different crops can be cultivated every year. Assume for simplicity that the income of 
the year is a deterministic function, (i, j) +» Ajj, depending only on the crop i of the preceding 
year, and of the crop j of the current year (a slightly more complex model in which the income 
of the year depends on the crops of the two preceding years is needed to explain the historical 
variations of crop rotations [Bac03]). The income of a sequence 1),... , i, of crops can be written 
as Cj, Aj,i, *** Aix_yi,» Where cj, is the income of the first year. The maximal income in k years is given 
by cA*—!b, where b = (1,... ,1). We next show an example. 


See) tte VN 
14 2 
A= 2 5° 7. 7 
ae ame OC qr Oe 2 

Here, vertices 1, 2, and 3 represent fallow (no crop), wheat, and oats, respectively. (We put no arc 
from 1 to 1, setting Aj; = —oo, to disallow two successive years of fallow.) The numerical values 
have no pretension to realism; however, the income ofa year of wheat is 11 after a year of fallow, this 
is greater than after a year of cereal (5 or 6, depending on whether wheat or oats was cultivated). 
An initial vector coherent with these data may be c = [—oo 11 8], meaning that the income of 
the first year is the same as the income after a year of fallow. We have cAb = 18, meaning that the 
optimal income in 2 years is 18. This corresponds to the optimal walk (2, 3), indicating that wheat 
and oats should be successively cultivated during these 2 years. 


The Maximal Cycle Mean 


Definitions: 


1. 


2 


Facts: 
1, 


The maximal cycle mean, Pmax(A), of a matrix A € R!*”, is the maximum of the weight-to-length 
ratio over all cycles c of (A), that is, 


Pmax(A) = —— = max max —~*—_—_—_“+, (25.1) 


Icla Ai, + 0++ + Aiiy 
Ic| KEL inject k 


max 
c cycle of P(A) 


Denote by R4*" the set of real n x n matrices with nonnegative entries. For A € R{*" and p > 0, 
A?) is by definition the matrix such that (Al?) )ij = (Aij)?, and 


A= OA 


where p denotes the (usual) spectral radius. We also define 0..(A) = limps +00 Pp(A). 


[CG79], [Gau92, Ch. IV], [BSvdD95] Max-plus Collatz—Wielandt formula, I. Let A € R'X” and 


max 


A € R. The following assertions are equivalent: (i) There exists u € R” such that Au < Au; (ii) 
Pmax(A) < X. It follows that 


Pmax(A) = inf max (Au); / u; 


ucR*" l<i<n 
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(the product Auand the division by u; should be understood in the max-plus sense). If #max(A) > O, 
then this infimum is attained by some u € R”. If in addition A is irreducible, then Assertion (i) is 
equivalent to the following: (i’) there exists u € R"_,, \ {0} such that Au < Au. 


max 


. [Gau92, Ch. IV], [BSvdD95] Max-plus Collatz—Wielandt formula, I. Let } € Rmax. The following 


assertions are equivalent: (i) There exists u € R"., \ {0} such that Au > Au; (Ii) Pmax(A) > A. It 
follows that 


max A)= i A i is 
pee neh \(0) ee i a 
uj 4O 
[Fri86] For A € R17", we have o.(A) = exp(Pmax(log(A))), where log is interpreted entrywise. 
[KO85] For all A € R{*", and 1 < q < p < 00, we have p,(A) < p,(A). 
For all A, B € RX", we have 


1 1 
p(Ao B) < p,(A)p,(B) forall p,q € [1,00] such that > +—-=1. 
q 


This follows from the classical Kingman’s inequality [Kin61], which states that the map log op o exp 
is convex (exp is interpreted entrywise). We have in particular p(A 0 B) < poo(A)p(B). 
[Fri86] For all A € R%*", we have 


Poo( A) < p(A) < Poo( A)p(A) < poo(A)n, 


where A is the pattern matrix of A, that is, Aij = lif Ajj A Oand Aij = Oif Ajj = 0. 

[Bap98], [EvdD99] For all A € R'*", we have limp+o0(Poo(A*))!/* = p(A). 

[CG79] Computing Pmax(A) by linear programming. For A € RiX", Pmax(A) is the value of the 
linear program 


infAst.dueR", WG,j)eF, Ajtuj <A+u;, 


where E = {(i,j) | 1<1i,j <n, Ajj; ¥ O} is the set of arcs of P(A). 
Dual linear program to compute Pmax(A). Let C denote the set of nonnegative vectors x = (Xjj)(i,j)ez 
such that 


V1 < 1 <n, y- XG = .s Xij> and ‘a Xij sls 


l<k<n, (ksi)€E l<j<n(ij)eE (ez 


To every cycle c of (A) corresponds bijectively the extreme point of the polytope C that is given by 
xij = 1/|cl if Gi, j) belongs toc, and x;; = 0 otherwise. Moreover, Pmax(A) = sup{)o(i,j)ek Aijxij | 
x EC}. 

[Kar78] Karp’s formula. If A € R"*” is irreducible, then, for all 1 <i <n, 


max 


a CAT) = (APs 
Pmax(A) = max min 7". 
At 40 


(25.2) 


To evaluate the right-hand side expression, compute the sequence uw =e,u!' = uw°A,u" =u"! A, 
so that ué = Ar for all 0 < k < n. This takes a time O(nm), where m is the number of arcs of 
T'(A). One can avoid storing the vectors u°,...,u”, at the price of recomputing the sequence 
u°,...,u"-! once u” is known. The time and space complexity of Karp’s algorithm are O(nm) 
and O(n), respectively. The policy iteration algorithm of [CTCG*98] seems experimentally more 
efficient than Karp’s algorithm. Other algorithms are given in particular in [CGL96], [BO93], 
and [EvdD99]. A comparison of maximal cycle mean algorithms appears in [DGI98]. When the 
entries of A take only two finite values, the maximal cycle mean of A can be computed in linear 
time [CGB95]. The Karp and policy iteration algorithms, as well as the general max-plus operations 
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(full and sparse matrix products, matrix residuation, etc.) are implemented in the Maxplus toolbox 
of Scilab, freely available in the contributed section of the Web site www.scilab.org. 


Example: 
1. For the matrix A in Application 3 of section 25.1, we have Pmax(A) = max(5, 4, (2 + 11)/2,(2+ 
8)/2, (7+6)/2, (11+7+2)/3, (8+6+2)/3) = 20/3, which gives the maximal reward per year. This 
is attained by the cycle (1, 2), (2, 3), (3, 1), corresponding to the rotation of crops: fallow, wheat, oats. 


25.3. The Max-Plus Eigenproblem 


The results of this section and of the next one constitute max-plus spectral theory. Early and fundamental 
contributions are due to Cuninghame-—Green (see [CG79]), Vorobyev [Vor67], Romanovskii [Rom67], 
Gondran and Minoux [GM77], and Cohen, Dubois, Quadrat, and Viot [CDQV83]. General presentations 
are included in [CG79], [BCOQ92], and [GM02]. The infinite dimensional max-plus spectral theory 
(which is not covered here) has been developed particularly after Maslov, in relation with Hamilton— 
Jacobi partial differential equations; see [MS92] and [KM97]. See also [MPN02], [AGW05], and [Fat06] 
for recent developments. 
In this section and the next two, A denotes a matrix in R?”. 


Definitions: 


An eigenvector of A is a vector u € R’.,, \ {0} such that Au = Au, for some scalar A € Rmax, which is 
called the (geometric) eigenvalue corresponding to u. With the notation of classical algebra, the equation 
Au = iu can be rewritten as 

max Ajj tuj=A+uj,, Wl<i<n. 

l<j<n 
If A is an eigenvalue of A, the set of vectors u € R’.,, such that Au = Au is the eigenspace of A for the 
eigenvalue i. 


The saturation digraph with respect to u € R".,, Sat(A, u), is the digraph with vertices 1,... ,m and 
an arc from vertex i to vertex j when Ajjuj = (Au);. 
Acyclec = (ij,12),... 5 (ig, 41) that attains the maximum in (25.1) is called critical. The critical digraph 


is the union of the critical cycles. The critical vertices are the vertices of the critical digraph. 
The normalized matrix is A = Pmax(A)~! A (when pmax(A) 4 0). 


Fora digraph I’, vertex has access to a vertex j ifthere is a walk from to j inl’. The (access equivalent) 
classes of I are the equivalence classes of the set of its vertices for the relation “i has access to j and j has 
access to i.” A class C has access to a class C’ if some vertex of C has access to some vertex of C’. A class is 
final if it has access only to itself. 


The classes of a matrix A are the classes of (A), and the critical classes of A are the classes of the 
critical digraph of A. A class C of A is basic if Pmax(A[C, C]) = Pmax(A). 


Facts: 


The proof of most of the following facts can be found in particular in [CG79] or [BCOQ92, Sec. 3.7]; we 
give specific references when needed. 


1. For any matrix A, Pmax(A) is an eigenvalue of A, and any eigenvalue of A is less than or equal to 
Pmax(A). 

2. An eigenvalue of A associated with an eigenvector in R” must be equal to Pmax(A). 

3. [ES75] Max-plus diagonal scaling. Assume that u € R" is an eigenvector of A. Then the matrix B 
such that Bj; = uj; 1A; ju, has all its entries less than or equal to Pmax(A), and the maximum of 
every of its rows is equal to Pmax(A). 

4. If A is irreducible, then pmax(A) > Oand it is the only eigenvalue of A. From now on, we assume 
that T'(A) has at least one cycle, so that Pmax(A) > O. 
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5. 


6. 


11. 
12. 


13. 


For all critical vertices i of A, the column At is an eigenvector of A for the eigenvalue Pmax(A). 
Moreover, if i and j belong to the same critical class of A, then A’, = At’ Ae, 

Eigenspace for the eigenvalue Pmax(A). LetCi,... , Cs denote the critical classes of A, and let us choose 
arbitrarily one vertex i; € C;, for every t = 1,...,s. Then, the columns At t= 1,...,s span 
the eigenspace of A for the eigenvalue Pmax(A). Moreover, any spanning family of this eigenspace 
contains some scalar multiple of every column At, a CR 

Let C denote the set of critical vertices, and let T = {1,... ,n}\ C. The following facts are proved 
in a more general setting in [AGO3, Th. 3.4], with the exception of (b), which follows from Fact 4 
of Section 25.1. 


(a) The restriction v +> v[C] is an isomorphism from the eigenspace of A for the eigenvalue 
Pmax(A) to the eigenspace of A[C, C] for the same eigenvalue. 

(b) An eigenvector u for the eigenvalue Pmax(A) is determined from its restriction u[C] by 
u(T] = (A[T, T])* A[T, C]u[C]. 

(c) Moreover, Pmax(A) is the only eigenvalue of A[C, C] and the eigenspace of A[C, C] is stable 
by infimum and by convex combination in the usual sense. 

Complementary slackness. If u € Ri, is such that Au < Pmax(A)u, then (Au); = Pmax(A)uj, for 

all critical vertices i. 

Critical digraph vs. saturation digraph. Let u € R" be such that Au < Pmax(A)u. Then, the union 

of the cycles of Sat(A, u) is equal to the critical digraph of A. 


. [CQD90], [Gau92, Ch. IV], [BSvdD95] Spectrum of reducible matrices. A scalar 1. # Ois an eigen- 


value of A if and only if there is at least one class C of A such that Pmax(A[C,C]) = A and 
Pmax(A[C,C]) > Pmax(A[C’, C’]) for all classes C’ that have access to C. 

[CQD90], [BSvdD95] The matrix A hasan eigenvector in R” ifand only ifall its final classes are basic. 
[Gau92, Ch. IV] Eigenspace for an eigenvalue h. Let C 1 ...,C™ denote all the classes C of A such 
that Pmax(A[C, C]) = A and Pmax(A[C’, C’]) < A for all classes C’ that have access to C. For every 
1<k<™m,let cr, easy Gc denote the critical classes of the matrix A[C*, C*]. For alll < k <m 
and 1 < t < sx, let us choose arbitrarily an element j,, in Ce Then, the family of columns 
GolAy 9 indexed by all these k and t, spans the eigenspace of A for the eigenvalue 1, and any 
spanning family of this eigenspace contains a scalar multiple of every (A~! A)* jes 

Computing the eigenvectors. Observe first that any vertex j that attains the maximum in Karp’s for- 
mula (25.2) is critical. To compute one eigenvector for the eigenvalue Pmax(A), it suffices to compute 
A*, for some critical vertex j. This is equivalent to a single source shortest path problem, which can 
be solved in O(nm) time and O(n) space. Alternatively, one may use the policy iteration algorithm 
of [CTCGt 98] or the improvement in [EvdD99] of the power algorithm [BO93]. Once a particular 
eigenvector is known, the critical digraph can be computed from Fact 9 in O(m) additional time. 


Examples: 


Ls 


For the matrix A in Application 3 of section 25.1, the only critical cycle is (1, 2), (2, 3), (3, 1) (up toa 
circular permutation of vertices). The critical digraph consists of the vertices and arcs of this cycle. 
By Fact 6, any eigenvector u of A is proportional to A* = [0 —13/3 —14/3]" (or equivalently, to A’, 
or A*,). Observe that an eigenvector yields a relative price information between the different states. 


. Consider the matrix and its associated digraph: 


Oo} - O -: 7 


ake a 
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(We use - to represent the element —oo.) The classes of A are C! = {1}, C? = {2,3,4}, C? = 
(5, 6, 7}, and c = {8}. We have Pmax( A) =; Ons AIC =) = 2, Pinax( A[C*, C')) = 0, 
Pmax(A[C?, C?]) = 1, and pmax(A[C*, C4]) = —3. The critical digraph is reduced to the crit- 
ical cycle (2,3)(3,2). By Fact 6, any eigenvector for the eigenvalue Pmax(A) is proportional to 
A’, = [—3 0 —1 0 —0o —o0 —00 —o0]". By Fact 10, the other eigenvalues of A are 0 and 1. By 
Fact 12, any eigenvector for the eigenvalue 0 is proportional to A*, = e;. Observe that the critical 
classes of A[C?, C*] are Cc} = {5} and Cc} = {6,7}. Therefore, by Fact 12, any eigenvector for the 
eigenvalue 1 is a max-plus linear combination of (17! A)*, = [6 —co —oo —00 0 —3 —2 oo]? 
and (17! A)* = [5 —oo —00 —00 -101 oo]. The eigenvalues of A? are 2,1, and —3. So Aand 
AT have only two eigenvalues in common. 


Asymptotics of Matrix Powers 


Definitions: 
A sequence so,5),... of elements of Rmax is recognizable if there exists a positive integer p, vectors 
1 1 LE _ k : * 

b e REX andc € R,X?, anda matrix M € RP”? such that s, = cM"b, for all nonnegative integers k. 

A sequence So,51,... of elements of Ryax is ultimately geometric with rate A € Rmax if sp41 = As, for 
k large enough. 

The merge of q sequences s!,...,s4 is the sequence s such that Skgti-1 = sh for all k > 0 and 
l<i<q. 
Facts: 

1. [Gun94], [CTGG99] If every row of the matrix A has at least one entry different from O, then, for 


alll <i < nandue€ R’, the limit 
xi(A) = lim (A‘u)}/* 
k->oo 
exists and is independent of the choice of u. The vector x(A) = (xi(A))i<i<n € R” is called the 
cycle-time of A. It is given by 
Xi (A) = max{Pmax(A[C, C]) | C is a class of A to which i has access}. 


In particular, if A is irreducible, then x;(A) = Pmax(A) for alli = 1,...,n. 


. The following constitutes the cyclicity theorem, due to Cohen, Dubois, Quadrat, and Viot [CDQV83]. 


See [BCOQ92] and [AGW05] for more accessible accounts. 


(a) If A is irreducible, there exists a positive integer y such that AK+Y = pinax(A)Y A* for k large 
enough. The minimal value of y is called the cyclicity of A. 

(b) Assume again that A is irreducible. Let C,...,C, be the critical classes of A, and fori = 
1,... ,5, let y; denote the g.c.d. (greatest common divisor) of the lengths of the critical cycles 
of A belonging to C;. Then, the cyclicity y of A is the lc.m. (least common multiple) of 
Vilo- ++ Vs- 

(c) Assume that Pmax(A) 4 O. The spectral projector of A is the matrix P := limy_..o Ak At = 
limg.0 A‘ + AM! 4.---. It is given by P = 3);-¢ A* At, where C denotes the set of critical 
vertices of A. When A is irreducible, the limit is attained in finite time. If, in addition, A has 
cyclicity one, then A¥ = pmax(A)* P for k large enough. 


. Assume that A is irreducible, and let m denote the number of arcs of its critical digraph. Then, the 


cyclicity of A can be computed in O(m) time from the critical digraph of A, using the algorithm 
of Denardo [Den77]. 
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4. The smallest integer k such that AMY = pmax(A)’ A* is called the coupling time. It is estimated 
in [HA99], [BGO1], [AGW05] (assuming again that A is irreducible). 

5. [AGW05, Th. 7.5] Turnpike theorem. Define a walk of P(A) to be optimal if it has a maximal weight 
amongst all walks with the same ends and length. If A is irreducible, then the number of noncritical 
vertices of an optimal walk (counted with multiplicities) is bounded by a constant depending only 
on A. 

6. [Mol88], [Gau94], [KB94], [DeSO00] A sequence of elements of Ryax is recognizable if and only if 
it is a merge of ultimately geometric sequences. In particular, for all 1 < i, 7 <n, the sequence 
(A*);; isa merge of ultimately geometric sequences. 

7. [Sim78], [Has90], [Sim94], [Gau96] One can decide whether a finitely generated semigroup S of 
matrices with effective entries in Rax is finite. One can also decide whether the set of entries in 
a given position of the matrices of S is finite (limitedness problem). However [Kro94], whether 
this set contains a given entry is undecidable (even when the entries of the matrices belong to 
ZU {—oo}). 


Example: 


1. For the matrix A in Application 3 of section 25.1, the cyclicity is 3, and the spectral projector is 


0 0 13/3 14/3 

oy T 
P= A‘ At = |-13/3| |0 13/3 14/3} = |-13/3 0 1/3 
—14/3 —14/3 -1/3 0 


2. For the matrix A in Example 2 of Section 25.3, the cycle-time is x(A) = [2222111 —-3]’. 
The cyclicity of A[C?,C7] is 2 because there is only one critical cycle, which has length 2. Let 
B := A[C?,C3]. The critical digraph of B has two strongly connected components consisting, 
respectively, of the cycles (5,5) and (6,7), (7,6). So B has cyclicity I.c.m. (1,2) = 2. The sequence 
5s t= (A*)j¢ is such that spy. = sp +4, for k > 24, with soq = 595 = 51. Hence, s; is the merge 
of two ultimately geometric sequences, both with rate 4. To get an example where different rates 
appear, replace the entries Aj; and Agg of A by —oo. Then, the same sequence s; is such that 
Se42 = Se +4, for all even k > 24, and sp42 = 5, +2, for all odd k > 5, with ss = 31 and s24 = 51. 


25.5 The Max-Plus Permanent 


Definitions: 


The (max-plus) permanent of A is per A = ae es, Alo(1)*** Ano(n), OF with the usual notation of classical 
algebra, per A = maxges, Aio(1) +- +: + Ano(n)> Which is the value of the optimal assignment problem 
with weights A;;. 

A max-plus polynomial function P is a map Rmax — Rmax of the form P(x) = yy pix! with 
Pi © Rmaxs i = 0,...,n. If py 4 O, P is of degree n. 

The roots of a nonzero max-plus polynomial function P are the points of nondifferentiability of P, 
together with the point O when the derivative of P near —oo is positive. The multiplicity of a root w of P 
is defined as the variation of the derivative of P at the point a, P’(a+) — P’(a~), when a ¥ O, and as its 
derivative near —oo, P’(O*), whena = O. 

The (max-plus) characteristic polynomial function of A is the polynomial function P,4 given by 
Pa(x) = per(A+ x1) for x € Rnax. The algebraic eigenvalues of A are the roots of P4. 


Facts: 


1. [CGM80] Any nonzero max-plus polynomial function P can be factored uniquely as P(x) = 
a(x -+a,)---(x-+ta@,), where a € R, n is the degree of P, and the a; are the roots of P, counted 
with multiplicities. 


25-10 Handbook of Linear Algebra 


2. [CG83], [ABG04, Th. 4.6 and 4.7]. The greatest algebraic eigenvalue of A is equal to Pmax(A). Its 
multiplicity is less than or equal to the number of critical vertices of A, with equality if and only if 
the critical vertices can be covered by disjoint critical cycles. 

3. Any geometric eigenvalue of A is an algebraic eigenvalue of A. (This can be deduced from Fact 2 
of this section, and Fact 10 of section 25.3.) 

4. [Yoe61] If A > I and per A = 1, then Ay; = per A(j,i), foralll <i,j <n. 

5. [But00] Assume that all the entries of A are different from O. The following are equivalent: (i) there 

is a vector b € R" that has a unique preimage by 4A; (ii) there is only one permutation o such that 

Ola := Ajo(1) +++ Ano(n) = per A. Further characterizations can be found in [But00] and [DSS05]. 

6. [Bap95] Alexandroff inequality over Rmax. Construct the matrix B with columns A., Ay, A3,..., 

A., and the matrix C with columns A, A», A3,..., A.». Then (per A)? > (per B)(per C), or 

with the notation of classical algebra, 2 x per A > per B + perC. 

7. [BB03] The max-plus characteristic polynomial function of A can be computed by solving O(n) 
optimal assignment problems. 


Example: 


1. For the matrix A in Example 2 of section 25.3, the characteristic polynomial of A is the product 
of the characteristic polynomials of the matrices A[(C',C'], fori = 1,...,4. Thus, Pa(x) = 
(x ++ 0)(x + 2)?x(x + 1)?(x +(—3)), and so, the algebraic eigenvalues of A are —oo, —3,0, 1, 
and 2, with respective multiplicities 1, 1, 1, 3, and 2. 


25.6 Linear Inequalities and Projections 


Definitions: 
IfAe re , the range of A, denoted range A, is {Ax | x € Re ahe Ri ox: The kernel of A, denoted 
ker A, is the set of equivalence classes modulo A, which are the classes for the equivalence relation “x ~ y 
if Ax = Ay.” 

The support of a vector b € Roiax is supp b := {i € {1,...,n} | b; 4 O}. 

The orthogonal congruence ofa subset U of R’.,,, is Uta = He y)ER. | u-x = u-y Vu € U}, 
eae “.” denotes the max-plus scalar product. The orthogonal space of a subset C of R”.,. x Ri.,, is 

={ueR. ju-x=u-y V(x y) € C}. 


max * Rinax 


max 


Facts: 


1. For alla, b € Rmax, the maximal c € Ryax such that ac < b, denoted by a \ b (or b / a), is given 
bya\b = b —aif (a,b) ¢ {(—ow, —o0), (400, +00)}, and a \ b = +00 otherwise. 

2. [BCOQ92, Eq. 4.82] If A € R.? and B € R. 4, then the inequation AX < B has a maximal 
solution X € Re"? given by the matrix A \ B defined by (A \ B)ij = Ai<ken Aki \ By;- Similarly, 
for A € Rn”? and C € R,,?, the maximal solution C / A € R..” of XA < C exists and is given 
by (C / A)ij = = Nik<p Cik / Aj. 

3. The equation - AX = B hasa solution if and only if A(A \ B) = 

4. For A € RR, the map A? : ye Ri, > A\ye R.., is ee It is represented by the 
matrix — A’. 

5. [BCOQ92, Table 4.1] For matrices A, B, C with entries in Ra, and with appropriate dimensions, 
we have 


A(A\(AB)) = AB, A\(A(A\ B)) = A\B, 
(A+ B)\C =(A\C)A(B\C), A\(BAC)=(A\B)A(A\C), 
(AB)\C=B\(A\C), A\(B/C)=(A\B)/C. 


Max-Plus Algebra 25-11 


10. 


11. 


12. 


13. 
14. 


15. 


16. 


by. 
18. 


. [CGQ04], [DS04] For any A € R* 


. [CGQ96], [CGQ97] Projection onto a range parallel to a kernel. Let B € R 


The first five identities have dual versions, with / instead of \\. Due to the last identity, we shall 
write A \ B / C instead of A \(B / C). 


. [CGQ97] Let A € RX?) B eR’! andC ¢€ R“?. We have rangeA C rangeeB => A= 


max ? max max * 


B(B\A),andkerA CkerC => C=(C/A)A. 


. [CGQ96] Let A € Re: The map Il, := Ao A® is a projector on the range of A, mean- 


ing that (14)? = I, and rangeI1, = range A. Moreover, I1,(x) is the greatest element of 
the range of A, which is less than or equal to x. Similarly, the map 14 := A® o A is a pro- 
jector on the range of A*, and I4(x) is the smallest element of the range of A® that is greater 
than or equal to x. Finally, every equivalence class modulo A meets the range of A* at a unique 
point. 

sey the map x > A(—x) is a bijection from range (A7) to 
range (A), with inverse map x b> A™ (=x). 

Re? and C € Rte". For 
allx € Brae there is a greatest € on the range of B such that CE < Cx. It is given by 1$(x), 

where IT p= Tlgp0 1°. We have a= Tg, Assume now that every equivalence class modulo 
C meets the range of B at a unique point. This is the case if and only if range (CB) = range C and 
ker(C B) = ker B. Then nS (x) is the unique element of the range of B, which is equivalent to x 
modulo C, the map I isa linear projector on the range of B, and it is represented by the matrix 
(B /(CB))C, which is equal to B((C B) \ C). 

[CGQ97] Regular matrices. Let A € R? . The following assertions are equivalent: (i) there 
is a linear projector from R ax to range A; (ii) A = AXA for some X € Re (iii) A = 
A(A\ A/ AJA 

[Vor67], [Zim76, Ch. 3] (See also [But94], [AGK05].) Vorobyev-Zimmermann covering theorem. 
Assume that A € R"*? andbe Ra x For j € {1,..., p}, let 


Sj = {i € {Ts se ,n} | Aij # Oand Aj; \b; = (A\b);}. 


The equation Ax = b has a solution if and only if U;<j<p5j; D> supp b or equivalently U jesupp(a\b) 
5; D supp b. It has a unique solution if and only if Ujesupp(a\b)$j; D supp band Uj; S$; B suppb 
for all strict subsets J of supp(A \ b). 

[Zim77], [SS92], [CGQ04], [CGQS05], [DS04] Separation theorem. Let A € R? andb € R... 

Ifb ¢ range A, then there exists c,d € Ri ox such that the halfspace H := {x € Rox |c-x >d-x} 
contains range A but not b. We can take c = —b and d = —I1,(b). Moreover, when A and b have 
entries in Ryax, ¢, d can be chosen with entries in Rypax. 

[GP97] For any A € R*? , we have ((range A)t)l = = range A. 

[LMS01], [CGQ04] A linear form defined on a finitely generated subsemimodule of Rix ¢ an 
be extended to R’.,.. This is a special case of a max-plus analogue of the Riesz representation 
theorem. 

[BH84], [GP97] Let A, B € R”.”?. The set of solutions x € R?.,, of Ax = Bxisa finitely generated 
subsemimodule of R? 

[GP97], [Gau98] Let X, Y be finitely generated subsemimodules of R. ae A€é R.. i and B € aa 

Then XN Y, X+Y := {x+y |x e€ X, y € Y}, and X —Y ee eke | dx e Xye€ 
Y, x = y+z} are finitely generated subsemimodules of Rr" Also, A -1(X), B(X), and X+ are 
finitely generated subsemimodules of Rinax? R, max» respectively. Similarly, if Z is a 
finitely generated subsemimodule of Ry... X Rinax> then Z rT i is a finitely generated subsemimodule 
of Ry ay: 
Facts 13 to 16 still hold if Rmx is replaced by Rinax- 

When A, B € R’*?, algorithms to find one solution of Ax = Bx are given in [WB98] or [CGBO03]. 
One can also use the general algorithm of [GG98] to compute a finite fixed point of a min-max 
function, together with the observation that x satisfies Ax = Bx if and only ifx = f(x), where 


f(x) = x (A\(Bx)) A (B \(Ax)). 


max" 


andR’. x XR, 


max? 


max? 
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FIGURE 25.1 Projection of a point on a range. 
Examples: 
1. In order to illustrate Fact 11, consider 
0 0 0 -oo 0.5 3 
Absa: (Ih 29> 4G 0 151, b= |o}. (25.3) 
0 3.2 0 3 0.5 


Let x := A\b. We have x} = min(—0 + 3,—1+0,—0+ 0.5) = —1, and so, S; = {2} because 

the minimum is attained only by the second term. Similarly, x. = —2.5, S. = {3}, %; = —1.5, 

S3 = {3}, X% = 0, Sy = {2}, X5 2.5, Ss = {3}. Since Uj <j<55; = {2,3} D suppb = {1, 2, 3}, 

Fact 11 shows that the equation Ax = b has no solution. This also follows from the fact that 

T14(b) = A(A\b) = [-100.5]? <b. 
2. The range of the previous matrix A is represented in Figure 25.1 (left). A nonzero vector x € R?,,. 
is represented by the point that is the barycenter with weights (exp(6x;))1<j<3 of the vertices of 
the simplex, where 6 > 0 is a fixed scaling parameter. Every vertex of the simplex represents one 
basis vector e;. Proportional vectors are represented by the same point. The i-th column of A, Aj, 
is represented by the point p; on the figure. Observe that the broken segment from p, to p2, which 
represents the semimodule generated by A., and A.2, contains ps. Indeed, A.5 = 0.5A.; + A.2. The 
range of A is represented by the closed region in dark grey and by the bold segments joining the 
points pj, pz, ps to it. 

We next compute a half-space separating the point b defined in (25.3) from range A. Recall 


that I1,(b) = [—1 0 0.5]. So, by Fact 12, a half-space containing range A and not b is H := 
{x € R. .(—3)x1 4h X) +(—0.5)x3 > 1x) + xX) +(—0.5)x3}. We also have HOR? = {x € R3,,, | 


X) +-(—0.5)x3 > 1x)}. The set of nonzero points of HMR3,,, is represented by the light gray region 
in Figure 25.1 (right). 


25.7 Max-Plus Linear Independence and Rank 


Definitions: 


If M is a subsemimodule of R’,,,, u € M is an extremal generator of M, or Rnaxu = {A.u| A € Rmax} is 
an extreme ray of M, if u 4 0 and if u = v+ w with v, w € M imply that u = vor u= w. 

A family u,,... , u, of vectors of Ri.,, is linearly independent in the Gondran—Minoux sense if for all 
disjoints subsets I and J of {1,...,r}, andall A; € Rmax,i € UJ, we have Se Ayujy Aj Uj 
unless A; = Oforallic IUJ. 


je 
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For A € R"*", we define 


max 


det* A:= >> Ajo(1) tee Ano(n)> det A:= > Ajo(1) tee Ano(n)> 


oeSi oeSy 


where S* and S> are, respectively, the sets of even and odd permutations of {1,...,n}. The 
bideterminant [GM84] of A is (det* A, det™ A). 


For A € R"*? \ {0}, we define 


max 


* The row rank (resp. the column rank) of A, denoted rk, w(A) (resp. rk.o1(A)), as the number of 
extreme rays of range A’ (resp. range A). 

* The Schein rank of A as rks,(A) := min{r > 1 | A= BC, with B e RVS,C € RiP}. 

* The strong rank of A, denoted rky(A), as the maximal r > 1 such that there exists an r x r 
submatrix B of A for which there is only one permutation o such that |o|g = per B. 

* The row (resp. column) Gondran—Minoux rank of A, denoted rkgy(A) (resp. rkgmc), as the 
maximal r > 1 such that A has r linearly independent rows (resp. columns) in the Gondran— 
Minoux sense. 

* The symmetrized rank of A, denoted rk.ym(A), as the maximal r > 1 such that A has anr x r 
submatrix B such that det* B 4 det™ B. 


(A new rank notion, Kapranov rank, which is not discussed here, has been recently studied [DSS05]. We 
also note that the Schein rank is called in this reference Barvinok rank.) 


Facts: 
1. [Hel88], [Mol88], [Wag91], [Gau98], [DS04] Let M bea finitely generated subsemimodule of R’ 


A subset of vectors of M spans M if and only if it contains at least one nonzero element of every 
extreme ray of M. 

2. [GM02] The columns of A € R’*” are linearly independent in the Gondran—Minoux sense if and 
only if det A A det™ A. 

3. [Plu90], [BCOQ92, Th. 3.78]. Max-plus Cramer’s formula. Let A € R"™", and let b-,b+ € R" 
Define the i-th positive Cramer’s determinant by 


Dj := dett (Ay... Aj—-ibt Ajai... An) det (Ay... Ajoib Ajai... An), 


and the i-th negative Cramer’s determinant, D; , by exchanging bt and b~ in the definition of D;*. 
Assume that xt,x” € R".,, have disjoint supports. Then Axt + b~ = Ax” ++b* implies that 


max 


(det* A)x* + (det” A)x; + D> = (det” A)x{ + (det* A)x; + D7 V1 <i <n. (25.4) 


The converse implication holds, and the vectors xt and x” are uniquely determined by (25.4), if 
dett A # det” A, and if Dt # D; or Dt = D; = 0, forall 1 <i <n. This result is formulated 
in a simpler way in [Plu90], [BCOQ92] using the symmetrization of the max-plus semiring, which 
leads to more general results. We note that the converse implication relies on the following semiring 
analogue of the classical adjugate identity: Aadj’ A+-det’ AI = Aadj A-++det* AI, where 
adj~ A := (det™ A(j,i))i<i,j<n. This identity, as well as analogues of many other determinantal 
identities, can be obtained using the general method of [RS84]. See, for instance, [GBCG98], where 
the derivation of the Binet—Cauchy identity is detailed. 
4. For A € R“*?, we have 


tkemr(A) Ttkrow(A) 
Ket = k, m = s k c < 
eee nee = ee STS ee 
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The second inequality follows from Fact 2, the third one from Facts 2 and 3. The other inequalities 
are immediate. Moreover, all these inequalities become equalities if A is regular [CGQ06]. 


Examples: 


1. The matrix A in Example 1 of section 25.6 has column rank 4: The extremal rays of range A are 
generated by the first four columns of A. All the other ranks of A are equal to 3. 
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Generalizations of the Perron—Frobenius theory of nonnegative matrices to linear operators leaving a 
cone invariant were first developed for operators on a Banach space by Krein and Rutman [KR48], Karlin 
[Kar59], and Schaefer [Sfr66], although there are early examples in finite dimensions, e.g., [Sch65] and 
[Bir67]. In this chapter, we describe a generalization that is sometimes called the geometric spectral theory 
of nonnegative linear operators in finite dimensions, which emerged in the late 1980s. Motivated by a search 
for geometric analogs of results in the previously developed combinatorial spectral theory of (reducible) 
nonnegative matrices (for reviews see [Sch86] and [Her99]), this area is a study of the Perron—Frobenius 
theory of a nonnegative matrix and its generalizations from the cone-theoretic viewpoint. The treatment 
is linear-algebraic and cone-theoretic (geometric) with the facial and duality concepts and occasionally 
certain elementary analytic tools playing the dominant role. The theory is particularly rich when the 
underlying cone is polyhedral (finitely generated) and it reduces to the nonnegative matrix case when the 
cone is simplicial. 


26.1 Perron—Frobenius Theorem for Cones 


We work with cones in a real vector space, as “cone” is a real concept. To deal with cones in C”, we can 
identify the latter space with R®”. For a discussion on the connection between the real and complex case 
of the spectral theory, see [TS94, Sect. 8]. 


Definitions: 


A proper cone K ina finite-dimensional real vector space V is a closed, pointed, full convex cone, viz. 


*K+KCK,viz.xyeK =x+yek. 
* R'K CK,viz.xe Kya €eR* = axe K. 


* K is closed in the usual topology of V. 96-1 
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* KN (—-K) = {0}, viz.x, -xe K => x=0. 
* intK + Q@, where intK is the interior of K. 


Usually, the unqualified term cone is defined by the first two items in the above definition. However, in 
this chapter we call a proper cone simply a cone. We denote by K a cone in R", n > 2. 

The vector x € R” is K -nonnegative, written x >* O,ifxe K. 

The vector x is K-semipositive, written x >* 0, ifx >* 0 andx 40. 

The vector x is K -positive, written x >* 0, ifx € int K. 

For x,y € R”, we write x >k y (x Sk y, x >K 
K -positive). 

The matrix A € R”"*" is K-nonnegative, written A >K oif AK CK. 

The matrix A is K-semipositive, written A >* 0, if A>“ Oand A 0. 

The matrix A is K -positive, written A > 0, if A(K \ {0}) C int K. 

For A, B € R"*", A>* B(A>* B, A>* B) means A— B > 0(A— B>* 0, A— B >¥ 0). 

A face F of a cone K C R” is a subset of K, which is a cone in the linear span of F such that 
xe F,x>k y= o> ye F. 

(In this chapter, F will always denote a face rather than a field, since the only fields involved are R and 
C.) Thus, F satisfies all definitions of a cone except that its interior may be empty. 

A face F of K isa trivial face if F = {0} or F=K. 

For a subset S ofa cone K, the intersection of all faces of K including S is called the face of K generated 
by S and is denoted by ®(S). If S = {x}, then ®(S) is written simply as ®(x). 

For faces F, G of K, their meet and join are given respectively by FAG = FNGandF VG = ®(FUG). 

Avectorx € K isan extreme vector if either x is the zero vector or xis nonzero and ®(x) = {Ax : A > 0}; 
in the latter case, the face ®(x) is called an extreme ray. 

If P is K-nonnegative, then a face F of K isa P-invariant face if PF C F. 

If P is K-nonnegative, then P is K-irreducible if the only P -invariant faces are the trivial faces. 

If K is a cone in R", then a cone, called the dual cone of K, is denoted and given by 


y) if x — y is K-nonnegative (K -semipositive, 


K* = {y eR": y’x> 0 forallx € K}. 


If Ais an n x n complex matrix and x is a vector in C”, then the local spectral radius of A at x is denoted 
and given by px(A) = lim supy-+00|| A”x||'/", where || - || is any norm of C”. For A € C”*", its spectral 
radius is denoted by p(A) (or p) (cf. Section 4.3). 


Facts: 
Let K be acone in R". 


1. The condition intK 4 @ in the definition of a cone is equivalent to K — K = V, viz., for allz € V 
there exist x,y € K such thatz = x — y. 

2. A K-positive matrix is K -irreducible. 

3. [Van68], [SV70] Let P be a K -nonnegative matrix. The following are equivalent: 


(a) P is K-irreducible. 

(b) 30% Pt 0, 

(c) 7+ P)"!5* 0. 

(d) No eigenvector of P (for any eigenvalue) lies on the boundary of K. 


4. (Generalization of Perron—Frobenius Theorem) [KR48], [BS75] Let P be a K -irreducible matrix 
with spectral radius p. Then 


(a) p is positive and is a simple eigenvalue of P. 

(b) There exists a (up to a scalar multiple) unique K -positive (right) eigenvector u of P corre- 
sponding to p. 

(c) wis the only K -semipositive eigenvector for P (for any eigenvalue). 

(d) KN (pI — P)R" = {0}. 
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5. (Generalization of Perron—Frobenius Theorem) Let P be a K-nonnegative matrix with spectral 
radius p. Then 


(a) p is an eigenvalue of P. 


(b) There is a K -semipositive eigenvector of P corresponding to p. 


6. If P, Q are K-nonnegative and Q K< P, then p(Q) < p(P). Further, if P is K-irreducible and 
QK< P, then p(Q) < p(P). 

7. P is K-nonnegative (K-irreducible) if and only if P? is K*-nonnegative (K*-irreducible). 

8. If Aisann x nm complex matrix and x is a vector in C", then the local spectral radius p,(A) of A at 
x is equal to the spectral radius of the restriction of A to the A-cyclic subspace generated by x, i.e., 
span{A'x :i=0,1,...}.Ifxis nonzero andx = x; +---+ x; is the representation of x as a sum of 
generalized eigenvectors of A corresponding, respectively, to distinct eigenvalues A;,... , Ax, then 
Px(A) is also equal to max; <;<x|Aj|- 

9. Barker and Schneider [BS75] developed Perron—Frobenius theory in the setting of a (possibly 
infinite-dimensional) vector space over a fully ordered field without topology. They introduced the 
concepts of irreducibility and strong irreducibility, and show that these two concepts are equivalent 
if the underlying cone has ascending chain condition on faces. See [ERS95] for the role of real 
closed-ordered fields in this theory. 


Examples: 
1. The nonnegative orthant (Rt )" in R” isacone. Thenx >* Oifand only ifx > 0, viz. the entries of x 
are nonnegative, and F isa face of (Ri )" ifand only if F is of the form F; forsome J C {1,...,n}, 
where 


Fy = {xe (Rf)" : x; =0,i1¢ J}. 
Further, P >* 0(P >* 0, P >¥* 0, P is K-irreducible) if and only if P > 0(P > 0, P > 0, P 
is irreducible) in the sense used for nonnegative matrices, cf. Chapter 9. 
2. The nontrivial faces of the Lorentz (ice cream) cone K,, in R", viz. 


Ky = {x © R: (x2 +--+ +221)? < xa, 


are precisely its extreme rays, each generated by a nonzero boundary vector, that is, one for which 
the equality holds above. The matrix 


is K3-irreducible [BP79, p. 22]. 


26.2 Collatz—Wielandt Sets and Distinguished 
Eigenvalues 


Collatz—Wielandt sets were apparently first defined in [BS75]. However, they are so-called because they are 
closely related to Wielandt’s proof of the Perron—Frobenius theorem for irreducible nonnegative matrices, 
[Wie50], which employs an inequality found in Collatz [Col42]. See also [Sch96] for further remarks 
on Collatz—Wielandt sets and related max-min and min-max characterizations of the spectral radius of 
nonnegative matrices and their generalizations. 
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Definitions: 

Let P be a K -nonnegative matrix. 

The Collatz—Wielandt sets associated with P ({BS75], [TW89], [TSO1], [TS03], and [Tam01]) are defined 
by 


Q(P)={w > 0: Axe K\{0}, Px >* wx}. 
Q,(P)={w > 0: axe int K, Px >* wx}. 
X(P)={o > 0: axe K\{0}, Px &< ox}. 
r(P)={o >0: dx e int K, Px ®< ox}. 


For a K -nonnegative vector x, the lower and upper Collatz—Wielandt numbers of x with respect to P are 
defined by 


rp(x)=sup {o>0: Px >* ex}, 
Rp(x) =inf {6 > 0: Px*< ox}, 


where we write Rp(x) = 00 if noo exists such that Px *< ox. 

A (nonnegative) eigenvalue of P is a distinguished eigenvalue for K if it has an associated 
K-semipositive eigenvector. 

The Perron space INEUP) (or N}) is the subspace consisting of all u € R” such that (P — pI ‘u=0 
for some positive integer k. (See Chapter 6.1 for a more general definition of N}(A).) 

If F isa P-invariant face of K, then the restriction of P to spanF is written as P| F. The spectral radius 
of P|F is written as p[F], and if A is an eigenvalue of P|F, its index is written as v,[F ]. 

A cone K in R” is polyhedral if it is the set of linear combinations with nonnegative coefficients of 
vectors taken from a finite subset of R”, and is simplicial if the finite subset is linearly independent. 


Facts: 


Let P bea K -nonnegative matrix. 


1. [TW89] A real number A is a distinguished eigenvalue of P for K if and only if A = pp»(P) for 
some K -semipositive vector b. 
2. [Tam90] Consider the following conditions: 


(a) p is the only distinguished eigenvalue of P for K. 

(b) x >* Oand Px X< px imply that Px = px. 

(c) The Perron space of P? contains a K *-positive vector. 
(d) p € Q)(P7). 


Conditions (a), (b), and (c) are always equivalent and are implied by condition (d). When K is 
polyhedral, condition (d) is also an equivalent condition. 
3. [Tam90] The following conditions are equivalent: 


(a) p(P) is the only distinguished eigenvalue of P for K and the index of p(P) is one. 
(b) For any vector x € R", Px K< (P)x implies that Px = p(P)x. 
(c) K N(plI — P)R" = {0}. 
(d) P? has a K*-positive eigenvector (corresponding to p(P)). 
4. [TW89] The following statements all hold: 


(a) [BS75] If P is K-irreducible, then 
sup Q(P) = sup Q,(P) = inf U(P) = inf Y|(P) = e(P). 


(b) sup Q(P) = inf X)(P) = p(P). 
(c) inf X(P) is equal to the least distinguished eigenvalue of P for K. 
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(d) sup Q,(P) = inf X(P") and, hence, is equal to the least distinguished eigenvalue of P? for 
K*. 
(e) sup Q(P) € Q(P) and inf X(P) € X(P). 


(f) When K is polyhedral, we have sup Q2)(P) € ()(P). For general cones, we may have 
sup Q)(P) ¢ Q,(P). 


(g) [Tam90] When K is polyhedral, p(P) € Q)(P) if and only if p(A) is the only distinguished 
eigenvalue of P? for K*. 


(h) [TSO3] p(P) € X(P) ifand only if ®((N)(P) 1 K)UC) = K, where C is the set {x € K : 
pPx(P) < p(P)} and Ni(P) is the Perron eigenspace of P. 


5. In the irreducible nonnegative matrix case, statement (b) of the preceding fact reduces to the 
well-known max-min and min-max characterizations of p(P) due to Wielandt. Schaefer [Sfr84] 
generalized the result to irreducible compact operators in L?-spaces and more recently Friedland 
[Fri90], [Fri91] also extended the characterizations in the settings ofa Banach space or a C*-algebra. 

6. [TW89, Theorem 2.4(i)] For any x >* 0,rp(x) < px(P) < Rp(x). (This fact extends the well- 
known inequality rp(x) < p(P) < Rp(x) in the nonnegative matrix case, due to Collatz [Col42] 
under the assumption that x is a positive vector and due to Wielandt [Wie50] under the assumption 
that P is irreducible and x is semipositive. For similar results concerning a nonnegative linear 
continuous operator in a Banach space, see [FN89].) 

7. A discussion on estimating p(P) or p,(P) by a convergent sequence of (lower or upper) Collatz— 
Wielandt numbers can be found in [TW89, Sect. 5] and [Tam01, Subsect. 3.1.4]. 

8. [GKT95, Corollary 3.2] If K is strictly convex (i.e., each boundary vector is extreme), then P has 
at most two distinguished eigenvalues. This fact supports the statement that the spectral theory of 
nonnegative linear operators depends on the geometry of the underlying cone. 


26.3 The Peripheral Spectrum, the Core, and 
the Perron—Schaefer Condition 


In addition to using Collatz—Wielandt sets to study Perron—Frobenius theory, we may also approach this 
theory by considering the core (whose definition will be given below). This geometric approach started 
with the work of Pullman [Pul71], who succeeded in rederiving the Frobenius theorem for irreducible 
nonnegative matrices. Naturally, this approach was also taken up in geometric spectral theory. It was found 
that there are close connections between the core, the peripheral spectrum, the Perron—Schaefer condition, 
and the distinguished faces ofa K -nonnegative linear operator. This led to a revival of interest in the Perron— 
Schaefer condition and associated conditions for the existence of a cone K such that a preassigned matrix 
is K-nonnegative. (See [Bir67], [Sfr66], [Van68], [Sch81].) The study has also led to the identification 
of necessary and equivalent conditions for a collection of Jordan blocks to correspond to the peripheral 
eigenvalues of a nonnegative matrix. (See [TS94] and [McD03].) The local Perron—Schaefer condition 
was identified in [TS01] and has played a role in the subsequent work. In the course of this investigation, 
methods were found for producing invariant cones for a matrix with the Perron—Schaefer condition, 
see [TS94], [Tam06]. These constructions may also be useful in the study of allied fields, such as linear 
dynamical systems. There invariant cones for matrices are often encountered. (See, for instance, [BNS89].) 


Definitions: 


If P is K-nonnegative, then a nonzero P-invariant face F of K is a distinguished face (associated with 
A) if for every P-invariant face G, with G C F, we have p[G] < p[F] (and p[F] =A). 

If A is an eigenvalue of A € C”*", then ker(A — AI)‘ is denoted by NE(A) fork = 1,2,..., the index of 
A is denoted by v,(A) (or v, when A is clear), and the generalized eigenspace at A is denoted by Nj (A). 
See Chapter 6.1 for more information. 
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Let Ae C”™". 

The order of a generalized eigenvector x for A is the smallest positive integer k such that (A—AI)‘x = 0. 
The maximal order of all K -semipositive generalized eigenvectors in N,’(A) is denoted by ord). 

The matrix A satisfies the Perron—Schaefer condition ([Sfr66], [Sch81]) if 


* p= p(A) isan eigenvalue of A. 
* Ifd is an eigenvalue of A and |A| = p, then v4(A) < va(p). 


If K isa cone and P is K -nonnegative, then the set Mo P'K, denoted by corex (P), is called the core 
of P relative to K. 

An eigenvalue i of A is called a peripheral eigenvalue if |1| = (A). The peripheral eigenvalues of A 
constitute the peripheral spectrum of A. 

Letx € C”. Then A satisfies the local Perron—Schaefer condition at x if there is a generalized eigenvector 
y of A corresponding to px(A) that appears as a term in the representation of x as a sum of generalized 
eigenvectors of A. Furthermore, the order of y is equal to the maximum of the orders of the generalized 
eigenvectors that appear in the representation and correspond to eigenvalues with modulus p,(A). 


Facts: 


1. [Sfr66, Chap. V] Let K be a cone in R” and let P be a K-nonnegative matrix. Then P satisfies the 

Perron—Schaefer condition. 

2. [Sch81] Let K be a cone in R” and let P be a K-nonnegative matrix with spectral radius p. Then 

P has at least m linearly independent K-semipositive eigenvectors corresponding to ¢, where m is 

the number of Jordan blocks in the Jordan form of P of maximal size that correspond to p. 

3. [Van68] Let A € R”*”. Then there exists a cone K in R” such that A is K-nonnegative if and only 

if A satisfies the Perron—Schaefer condition. 

4. [TS94] Let A € R”™" that satisfies the Perron—Schaefer condition. Let m be the number of Jordan 
blocks in the Jordan form of A of maximal size that correspond to p(A). Then for each positive 
integer k,m <k < dim N (A), there exists a cone K in R” such that A is K -nonnegative and dim 
span(N)(A) AK)=k. 

5. Let A € R"*". Let k be a nonnegative integer and let w;(A) consist of all linear combinations with 


nonnegative coefficients of Ak, Ak+1, |... The closure of w;(A) is a cone in its linear span if and 
only if A satisfies the Perron—Schaefer condition. (For this fact in the setting of complex matrices 
see [Sch81].) 

6. Necessary and sufficient conditions involving @; (A) so that A € C”*” has a positive (nonnegative) 
eigenvalue appear in [Sch81]. For the corresponding real versions, see [Tam06]. 

7. [Pul71], [TS94] If K is a cone and P is K-nonnegative, then corex (P) is a cone in its linear span 
and P(corex(P)) = corex (P). Furthermore, corex (P) is polyhedral (or simplicial) whenever K 
is. So when corex (P) is polyhedral, P permutes the extreme rays of corex (P). 

8. For a K-nonnegative matrix P, a characterization of K-irreducibility (as well as K-primitivity) of 
P in terms of corex(P), which extends the corresponding result of Pullman for a nonnegative 
matrix, can be found in [TS94]. 

9. [Pul71] If P is an irreducible nonnegative matrix, then the permutation induced by P on the 
extreme rays of core(p+)(P) is a single cycle of length equal to the number of distinct peripheral 
eigenvalues of P. (This fact can be regarded as a geometric characterization of the said quantity 
(cf. the known combinatorial characterization, see Fact 5(c) of Chapter 9.2), whereas part (b) of 
the next fact is its extension.) 

10. [TS94, Theorem 3.14] For a K-nonnegative matrix P, if corex(P) is a nonzero simplicial cone, 
then: 


(a) There is a one-to-one correspondence between the set of distinguished faces associated with 
nonzero eigenvalues and the set of cycles of the permutation tp induced by P on the extreme 
rays of corex (P). 


Matrices Leaving a Cone Invariant 26-7 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


20. 


(b) Ifo is a cycle of the induced permutation tp, then the peripheral eigenvalues of the restric- 
tion of P to the linear span of the distinguished P-invariant face F corresponding to o are 
simple and are exactly p[F] times all the d,th roots of unity, where d, is the length of the 
cycle o. 


[TS94] If P is K-nonnegative and corex (P) is nonzero polyhedral, then: 


(a) corex (P) consists of all linear combinations with nonnegative coefficients of the distinguished 
eigenvectors of positive powers of P corresponding to nonzero distinguished eigenvalues. 


(b) corex(P) does not contain a generalized eigenvector of any positive powers of P other than 
eigenvectors. 


This fact indicates that we cannot expect that the index of the spectral radius of a nonnegative linear 
operator can be determined from a knowledge of its core. 

A complete description of the core of a nonnegative matrix (relative to the nonnegative orthant) 
can be found in [TS94, Theorem 4.2]. 

For A € R"*", in order that there exists a cone K in R” such that AK = K and A hasa K -positive 
eigenvector, it is necessary and sufficient that A is nonzero, diagonalizable, all eigenvalues of A are 
of the same modulus, and (A) is an eigenvalue of A. For further equivalent conditions, see [TS94, 
Theorem 5.9]. 

For A € R"*", an equivalent condition given in terms of the peripheral eigenvalues of A so that 
there exists a cone K in R" such that A is K -nonnegative and (a) K is polyhedral, or (b) coreg (A) 
is polyhedral (simplicial or a single ray) can be found in [TS94, Theorems 7.9, 7.8, 7.12, 7.10]. 
[TS94, Theorem 7.12] Let A € R”*" with o(A) > 0 that satisfies the Perron—Schaefer condition. 
Let S denote the multiset of peripheral eigenvalues of A with maximal index (ie., v4(p)), the 
multiplicity of each element being equal to the number of corresponding blocks in the Jordan 
form of A of order v4(p). Let T be the multiset of peripheral eigenvalues of A for which there are 
corresponding blocks in the Jordan form of A of order less than v,4(), the multiplicity of each 
element being equal to the number of such corresponding blocks. The following conditions are 
equivalent: 


(a) There exists a cone K in R” such that A is K -nonnegative and corex (A) is simplicial. 


(b) There exists a multisubset T of T such that S U T is the multiset union of certain complete 
sets of roots of unity multiplied by p(A). 


McDonald [McD03] refers to the condition (b) that appears in the preceding result as the Tam— 
Schneider condition. She also provides another condition, called the extended Tam—Schneider 
condition, which is necessary and sufficient for a collection of Jordan blocks to correspond to the 
peripheral spectrum of a nonnegative matrix. 

[TSO1] If P is K-nonnegative and x is K -semipositive, then P satisfies the local Perron—Schaefer 

condition at x. 

[Tam06] Let A be ann x n real matrix, and let x be a given nonzero vector of R". The following 

conditions are equivalent : 

(a) A satisfies the local Perron—Schaefer condition at x. 

(b) The restriction of A to span{A'x :i=0,1,...} satisfies the Perron—Schaefer condition. 

(c) For every (or, for some) nonnegative integer k, the closure of a (A, x), where @; (A, x) consists 
of all linear combinations with nonnegative coefficients of Akx, Akt!x,..., is a cone in its 
linear span. 

(d) There is a cone C in a subspace of R” containing x such that AC C C. 

The local Perron—Schaefer condition has played a role in the work of [TSO1], [TS03], and [Tam04]. 

Further work involving this condition and the cones @,;(A,x) (defined in the preceding fact) will 

appear in [Tam06]. 

One may apply results on the core of a nonnegative matrix to rederive simply many known results 

on the limiting behavior of Markov chains. An illustration can be found in [Tam01, Sec. 4.6]. 
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26.4 Spectral Theory of K-Reducible Matrices 


In this section, we touch upon the geometric version of the extensive combinatorial spectral theory of 
reducible nonnegative matrices first found in [Fro12, Sect. 11] and continued in [Sch56]. Many subsequent 
developments are reviewed in [Sch86] and [Her99]. Results on the geometric spectral theory of reducible 
K -nonnegative matrices may be largely found in a series of papers by B.S. Tam, some jointly with Wu and 
H. Schneider ([TW89], [Tam90], [TS94], [TSO1], [TS03], [Tam04]). For a review containing considerably 
more information than this section, see [Tam01]. 

In some studies, the underlying cone is lattice-ordered (for a definition and much information, see 
[Sfr74]) and, in some studies, the Frobenius form of a reducible nonnegative matrix is generalized; 
see the work by Jang and Victory [JV93] on positive eventually compact linear operators on Banach 
lattices. However in the geometric spectral theory the Frobenius normal form of a nonnegative reducible 
matrix is not generalized as the underlying cone need not be lattice-ordered. Invariant faces are considered 
instead of the classes that play an important role in combinatorial spectral theory of nonnegative matrices; 
in particular, distinguished faces and semidistinguished faces are used in place of distinguished classes and 
semidistinguished classes, respectively. (For definitions of the preceding terms, see [TSO1].) 

It turns out that the various results on a reducible nonnegative matrix are extended to a K -nonnegative 
matrix in different degrees of generality. In particular, the Frobenius—Victory theorem ([Fro12], [Vic85]) 
is extended to a K -nonnegative matrix on a general cone. The following are extended to a polyhedral cone: 
The Rothblum index theorem ([Rot75]), a characterization (in terms of the accessibility relation between 
basic classes) for the spectral radius to have geometric multiplicity 1, for the spectral radius to have index 1 
([Sch56]), and a majorization relation between the (spectral) height characteristic and the (combinatorial) 
level characteristic of a nonnegative matrix ([HS91b]). Various conditions are used to generalize the 
theorem on equivalent conditions for equality of the two characteristics ([RiS78], [HS89], [HS91a]). Even 
for polyhedral cones there is no complete generalization for the nonnegative-basis theorem, not to mention 
the preferred-basis theorem ([Rot75], [RiS78], [Sch86], [HS88]). There is a natural conjecture for the latter 
case ([Tam04]). The attempts to carry out the extensions have also led to the identification of important 
new concepts or tools. For instance, the useful concepts of semidistinguished faces and of spectral pairs 
of faces associated with a K -nonnegative matrix are introduced in [TSO1] in proving the cone version of 
some of the combinatorial theorems referred to above. To achieve these ends certain elementary analytic 
tools are also brought in. 


Definitions: 


Let P bea K -nonnegative matrix. 

A nonzero P-invariant face F is a semidistinguished face if F contains in its relative interior a gen- 
eralized eigenvector of P and if F is not the join of two P-invariant faces that are properly included in 
F. 

A K-semipositive Jordan chain for P of length m (corresponding to p(P)) is a sequence of m 
K -semipositive vectors x, (P — p(P)I)x,... ,(P — p(P)I)™~!x such that (P — p(P)I)"x = 0. 

A basis for N>(P) is called a K -semipositive basis if it consists of K -semipositive vectors. 

A basis for N?(P) is called a K -semipositive Jordan basis for P if it is composed of K -semipositive 
Jordan chains for P. 

The set C(P, K) = {x € K : (P — p(P)I)'x € K forall positive integers i} is called the spectral cone 
of P (for K corresponding to p(P)). 

Denote v, by v. 

The height characteristic of P is the v-tuple 7(P) = (71,...,7,) given by: 


nk = dim(Nk(P)) — dim(NE“!(P)). 
The level characteristic of P is the v-tuple A(P) = (Aj,... Ay) given by: 


A, = dim span(Nk(P) OK) —dim span(Ns~'(P) NK). 
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The peak characteristic of P is the v-tuple &(P) = (&,...,&,) given by: 
& = dim(P — p(P)1)S"(NE NK). 


If A € C"*" and x is a nonzero vector of C”, then the order of x relative to A, denoted by ord,(x), 
is defined to be the maximum of the orders of the generalized eigenvectors, each corresponding to an 
eigenvalue of modulus p,(A) that appear in the representation of x as a sum of generalized eigenvectors 
of A. 

The ordered pair (x(A), ord,4(x)) is called the spectral pair of x relative to A and is denoted by sp (x). 
We also set sp4(0) = (0, 0) to take care of the zero vector 0. 

We use < to denote the lexicographic ordering between ordered pairs of real numbers, i.e., (a,b) X (c,d) 
if either a < c,ora =c andb < d. Incase (a,b) < (c,d) but (a,b) $ (c,d), we write (a,b) < (c,d). 


Facts: 

1. If A € C"*" and x is a vector of C”, then ord,(x) is equal to the size of the largest Jordan block 
in the Jordan form of the restriction of A to the A-cyclic subspace generated by x for a peripheral 
eigenvalue. 

Let P bea K -nonnegative matrix. 

2. In the nonnegative matrix case, the present definition of the level characteristic of P is equivalent 
to the usual graph-theoretic definition; see [NS94, (3.2)] or [Tam04, Remark 2.2]. 

3. [TSO1] For any x € K, the smallest P-invariant face containing x is equal to ®(X), where X = 
(1 + P)"~!x. Furthermore, spp (x) = spp(X). In the nonnegative matrix case, the said face is also 
equal to F;, where F; is as defined in Example 1 of Section 26.1 and J is the union of all classes of 
P having access to supp(x) = {i : x; > 0}. (For definitions of classes and the accessibility relation, 
see Chapter 9.) 

4. [TSO1] For any face F of K, P-invariant or not, the value of the spectral pair sp p (x) is independent 
of the choice of x from the relative interior of F . This common value, denoted by sp4(F), is referred 
to as the spectral pair of F relative to A. 

5. [TSO1] For any faces F, G of K, we have 


(a) spp(F) = spp (F), where F is the smallest P-invariant face of K, including F. 


(b) If F C G, then spp(F) ~ spp(G). If F, G are P-invariant faces and F C G, then spp(F) x 
spp(G); viz. either p[ F] < p[G] or p[F] = p[G] and v,jr)[F] < vpfq[G]. 


6. [TSO1] If K isacone with the property that the dual cone of each ofits faces is a facially exposed cone, 
for instance, when K isa polyhedral cone, a perfect cone, or equals P (n) (see [TSO1] for definitions), 
then for any nonzero P -invariant face G, G is semidistinguished if and only if spp(F) ~ spp(G) 
for all P-invariant faces F properly included in G. 

7. [Tam04] (Cone version of the Frobenius—Victory theorem, [Fro12], [Vic85], [Sch86]) 


(a) For any real number A, A is a distinguished eigenvalue of P if and only if A = p[F] for some 
distinguished face F of K. 

(b) If F is a distinguished face, then there is (up to multiples) a unique eigenvector x of P corre- 
sponding to p[F ] that lies in F. Furthermore, x belongs to the relative interior of F. 

(c) For each distinguished eigenvalue A of P, the extreme vectors of the cone N)(P)MK are precisely 
all the distinguished eigenvectors of P that lie in the relative interior of certain distinguished faces 
of K associated with i. 


8. Let P beanonnegative matrix. The Jordan form of P contains only one Jordan block corresponding 
to p(P) if and only if any two basic classes of P are comparable (with respect to the accessibility 
relation); all Jordan blocks corresponding to p(P) are of size 1 if and only if no two basic classes 
are comparable ([Schn56]). An extension of these results to a K -nonnegative matrix on a class of 
cones that contains all polyhedral cones can be found in [TSO1, Theorems 7.2 and 7.1]. 
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[Tam90, Theorem 7.5] If K is polyhedral, then: 


(a) There is a K-semipositive Jordan chain for P of length v,; thus, there is a K -semipositive 
vector in NUP} of order vp, viz. ord, = Vp. 


(b) The Perron space Nz (P) has a basis consisting of K -semipositive vectors. 


However, when K is nonpolyhedral, there need not exist a K -semipositive vector in N73 (P) of 
order v,, viz. ord, < v,. For a general distinguished eigenvalue 2, we always have ord, < v,, no 
matter whether K is polyhedral or not. 

Part (b) of the preceding fact is not yet a complete cone version of the nonnegative-basis theorem, as 
the latter theorem guarantees the existence ofa basis for the Perron space that consists of semipositive 
vectors that satisfy certain combinatorial properties. For a conjecture on a cone version of the 
nonnegative-basis theorem, see [Tam04, Conj. 9.1]. 

[TSO1, Theorem 5.1] (Cone version of the (combinatorial) generalization of the Rothblum index 
theorem, [Rot75], [HS88]). 

Let K bea polyhedral cone. Let A be a distinguished eigenvalue of P for K. Then there is a chain 
Fi C Fy Cc... C Fy of k = ord, distinct semidistinguished faces of K associated with A, but 
there is no such chain with more than ord, members. When K is a general cone, the maximum 
cardinality of a chain of semidistinguished faces associated with a distinguished eigenvalue A may 
be less than, equal to, or greater than ord); see [TSO1, Ex. 5.3, 5.4, 5.5]. 

For K = (R})", viz. P isa nonnegative matrix, characterizations of different types of P -invariant 
faces (in particular, the distinguished and semidistinguished faces) are given in [TSO1] (in terms of 
the concept of an initial subset for P; see [HS88] or [TS01] for definition of an initial subset). 
[Tam04] The spectral cone C(P, K) is always invariant under P — p(P)I and satisfies: 


N,(P)NK CC(A,K) © NU(P)OK. 


If K is polyhedral, then C(A, K) is a polyhedral cone in N; (P). 

(Generalization of corresponding results on nonnegative matrices, [NS94]) We always have &(P) < 
n(P) and &(P) < Ax (P) fork = 1,..., Vp. 

[Tam04, Theorem 5.9] Consider the following conditions : 


(a) n(P) =A(P). 

(b) n(P) = &(P). 

c) For eachk,k =1,..., vp, Ni(P) contains a K -semipositive basis. 

d) There exists a K -semipositive Jordan basis for P. 

e) Foreachk,k = 1,... ,v,, Ni (P) has a basis consisting of vectors taken from NPAC; K). 


f) For eachk,k = 1,... ,v,, we have 
nx(P) = dim(P — p(P)I)'![NS(P)NC(P, K)]. 


Conditions (a) to (c) are equivalent and so are conditions (d) to (f). Moreover, we always have 
(a)==(d), and when K is polyhedral, conditions (a) to (f) are all equivalent. 

As shown in [Tam04], the level of a nonzero vector x € Ni(P) can be defined to be the smallest 
positive integer k such that x € span(Ni(P) 1 K); when there is no such k the level is taken to 
be oo. Then the concepts of K -semipositive level basis, height-level basis, peak vector, etc., can be 
introduced and further conditions can be added to the list given in the preceding result. 

[Tam04, Theorem 7.2] If K is polyhedral, then A(P) = n(P). 

Cone-theoretic proofs for the preferred-basis theorem for a nonnegative matrix and for a result 
about the nonnegativity structure of the principal components of a nonnegative matrix can be 
found in [Tam04]. 
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26.5 Linear Equations over Cones 


Given a K -nonnegative matrix P anda vectorb € K, in this section we consider the solvability of following 
two linear equations over cones and some consequences: 


(AI — P)x=b, xe K (26.1) 
and 
(P —iI)x=b, xe K. (26.2) 


Equation (26.1) has been treated by several authors in finite-dimensional as well as infinite-dimensional 
settings, and several equivalent conditions for its solvability have been found. (See [TS03] for a detailed 
historical account.) The study of Equation (26.2) is relatively new. A treatment of the equation by graph- 
theoretic arguments for the special case when A = p(P) and K = (R¢)" can be found in [TW89]. The 
general case is considered in [TS03]. It turns out that the solvability of Equation (26.2) is a more delicate 
problem. It depends on whether i is greater than, equal to, or less than pp(P). 


Facts: 


Let P bea K -nonnegative matrix, let 0 4 b € K, and let A be a given positive real number. 


1. [TS03, Theorem 3.1] The following conditions are equivalent: 


(a) Equation (26.1) is solvable. 

(b) po(P) <A. 

(c) lim S* A~/ P‘b exists. 
mM OO Fo 

(d) lim (A7!P)™b = 0. 
m—>oo 

(e) (z,b) = 0 for each generalized eigenvector z of P’ corresponding to an eigenvalue with 
modulus greater than or equal to A. 

(f) (z,b) = 0 for each generalized eigenvector z of P? corresponding to a distinguished eigenvalue 
of P for K that is greater than or equal to A. 


2. For a fixed A, the set (AI — P)K M K, which consists of precisely all vectors b € K for which 
Equation (26.1) has a solution, is equal to {b € K : pp(P) < A} and isa face of K. 

3. For a fixed A, the set (P — AI)K M K, which consists of precisely all vectors b € K for which 
Equation (26.2) has a solution, is, in general, not a face of K. 

4. [TS03, Theorem 4.1] WhenA > pp(P), Equation (26.2) is solvable ifand only if A is a distinguished 
eigenvalue of P for K andb € ®(NJ(P)NK). 

5. [TSO3, Theorem 4.5] When A = pp(P), if Equation (26.2) is solvable, then 
be (P — pp (P)I)® 
(Ni(p)(P) 1 K). 

6. [TSO3, Theorem 4.19] Let r denote the largest real eigenvalue of P less than p(P). (If no such 
eigenvalues exist, take r = —oo.) Then for any 4, r < 4 < p(P), we have 


O((P —AI)DKNK)= O(N, (P)NK). 


Thus, a necessary condition for Equation (26.2) to have a solution is that b K< u for some 
ue Ni(P) OK. 

7. [TSO3, Theorem 5.11] Consider the following conditions: 
(a) p(P) € &\(P*). 
(b) No(P) AK= Nj(P) OK, and P has no eigenvectors in B(N}(P) A K) corresponding to an 
eigenvalue other than p(P). 
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(c) K N(P — p(P)I)K = {0} (equivalently, x >* 0, Px >* p(P)x imply that Px = p(P)x). 
We always have (a)==> (b)==>(c). When K is polyhedral, conditions (a), (b), and (c) are equivalent. 
When K is nonpolyhedral, the missing implications do not hold. 


26.6 Elementary Analytic Results 


In geometric spectral theory, besides the linear-algebraic method and the cone-theoretic method, certain 
elementary analytic methods have also been called into play; for example, the use of Jordan form or the 
components of a matrix. This approach may have begun with the work of Birkhoff [Bir67] and it was 
followed by Vandergraft [Van68] and Schneider [Sch81]. Friedland and Schneider [FS80] and Rothblum 
[Rot81] have also studied the asymptotic behavior of the powers of a nonnegative matrix, or their variants, 
by elementary analytic methods. The papers [TS94] and [TSO1] in the series also need a certain kind 
of analytic argument in their proofs; more specifically, they each make use of the K-nonnegativity of 
a certain matrix, either itself a component or a matrix defined in terms of the components of a given 
K -nonnegative matrix (see Facts 3 and 4 in this section). In [HNR90], Hartwig, Neumann, and Rose offer 
a (linear) algebraic-analytic approach to the Perron—Frobenius theory of a nonnegative matrix, one which 
utilizes the resolvent expansion, but does not involve the Frobenius normal form. Their approach is further 
developed by Neumann and Schneider ({NS92], [NS93], [NS94]). By employing the concept of spectral 
cone and combining the cone-theoretic methods developed in the earlier papers of the series with this 
algebraic-analytic method, Tam [Tam04] offers a unified treatment to reprove or extend (or partly extend) 
several well-known results in the combinatorial spectral theory of nonnegative matrices. The proofs given 
in [Tam04] rely on the fact that if K is a cone in R", then the set w(K) that consists of all K -nonnegative 
matrices is a cone in the matrix space R"*" and if, in addition, K is polyhedral, then so is 7(K) ([Fen53, 
p. 22], [SV70], [Tam77]). See [Tam01, Sec. 6.5] and [Tam04, Sec. 9] for further remarks on the use of the 
cone 2(K) in the study of the spectral properties of K -nonnegative matrices. 

In this section, we collect a few elementary analytic results (whose proofs rely on the Jordan form), which 
have proved to be useful in the study of the geometric spectral theory. In particular, Facts 3, 4, and 5 identify 
members of (K). As such, they can be regarded as nice results, which are difficult to come by for the 
following reason: If K is nonsimplicial, then z(K ) must contain matrices that are not nonnegative linear 
combinations of its rank-one members ([Tam77]). However, not much is known about such matrices 
([Tam92]). 


Definitions: 


Let P bea K -nonnegative matrix. Denote v, by v. 

The principal eigenprojection of P, denoted by Z San is the projection of C” onto the Perron space N? 
along the direct sum of other generalized eigenspaces of P. 

For k = 0,... ,v, the kth principal component of P is given by 


ZS (P= (PZ: 


The kth component of P corresponding to an eigenvalue A is defined in a similar way. 
For k = 0,... , ¢, the kth transform principal component of P is given by: 


Te) = 7 + Zar ie feet Zap ate) for all ¢ € C\ {0}. 


Facts: 
Let P bea K-nonnegative matrix. Denote v, by v. 
1. [Kar59], [Sch81] Zoe is K -nonnegative. 


2. [TS94, Theorem 4.19(i)] The sum ofthe vth components of P corresponding to its peripheral eigen- 
values is K -nonnegative; it is the limit of a convergent subsequence of ((v — L)!PE/[pk-*t1ke]), 
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3. [Tam04, Theorem 3.6(i)] If K is a polyhedral cone, then for k = 0,...,v — 1, Tee) is 
K -nonnegative for all sufficiently small positive e. 


26.7 Splitting Theorems and Stability 


Splitting theorems for matrices have played a large role in the study of convergence of iterations in numerical 
linear algebra; see [Var62]. Here we present a cone version ofa splitting theorem which is proven in [Sch65] 
and applied to stability (inertia) theorems for matrices. A closely related result is generalized to operators 
ona partially ordered Banach space in [DH03] and [Dam04]. There it is used to describe stability properties 
of (stochastic) control systems and to derive non-local convergence results for Newton’s method applied 
to nonlinear operator equation of Riccati type. We also discuss several kinds of positivity for operators 
involving a cone that are relevant to the applications mentioned. 

For recent splitting theorems involving cones, see [SSA05]. For applications of theorems of the alternative 
for cones to the stability of matrices, see [CHS97]. Cones occur in many parts of stability theory; see, for 
instance, [Her98]. 


Definitions: 


Let K bea cone in R" and let A € R"*”. 
A is positive stable if spec(A) © C* viz. the spectrum of A is contained in the open right half-plane. 
A is K -inverse nonnegative if A is nonsingular and A7! is K -nonnegative. 
A is K-resolvent nonnegative if there exists an a € R such that, for all aw > a, aI — Ais K-inverse 
nonnegative. 
Ais cross-positive on K if for allx ¢ K,y € K*, y’x = 0 implies y? Ax > 0. 
A is a Z-matrix if all of its off-diagonal entries are nonpositive. 


Facts: 
Let K beacone in R". 

1. Ais K-resolvent nonnegative if and only if A is cross-positive on K. Other equivalent conditions 
and also Perron-Frobenius type theorems for the class of cross-positive matrices can be found in 
[Els74], [SV70] or [BNS89]. 

2. When K is (IRj)", A is cross-positive on K if and only if —A is a Z-matrix. 

3. [Sch65], [Sch97]. Let T = R — P where R, P € R"*” and suppose that P is K -nonnegative. If R 
satisfies R(intK) D intK or R(intK) MN intK = @, then the following are equivalent: 

(a) T is K-inverse nonnegative. 

(b) For all y >* 0 there exists (unique) x >* 0 such that y = Tx. 
(c) There exists x >* 0 such that Tx >* 0. 

(d) There exists x >* 0 such that Tx >* 0. 

(e) Ris K-inverse nonnegative and p(R7'P) <1. 


4, Let T € R"". If —T is K -resolvent nonnegative, then T satisfies T(intK ) > intK or 
T(intK) MN intK = 0. But the converse is false, see Example 1 below. 

5. [DH03, Theorem 2.11], [Dam04, Theorem 3.2.10]. Let T, R, P be as given in Fact 3. If —R is 
K -resolvent nonnegative, then conditions (a)—(e) of Fact 3 are equivalent. Moreover, the following 
are additional equivalent conditions: 

(f) T is positive stable. 
(g) R is positive stable and p(R7'P) <1. 


6. If K is (Rt)", R =a and P is a nonnegative matrix, then T = R — P is a Z-matrix. It satisfies 
the equivalent conditions (a)—(g) of Facts 3 and 5 ifand only if it is an M-matrix [BP79, Chapter 6]. 
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7. (Special case of Fact 5 with P = 0). Let T € R”*". If —T is K-resolvent nonnegetive, then 
conditions (a)—(d) of Fact 3 and conditions (f) of Fact 5 are equivalent. 

8. In [GT06] a matrix T is called a Z-tranformation on K if —T is cross-positive on K. Many 
properties on Z-matrices, such as being a P-matrix, a Q-matrix (which has connection with 
the linear complementarity problem), an inverse-nonnegative matrix, a positive stable matrix, a 
diagonally stable matrix, etc., are extended to Z-transformations. For a Z-transformation, the 
equivalence of these properties is examined for various kinds of cones, particularly for symmetric 
cones in Euclidean Jordan algebras. 

9. [Schn65], [Schn97]. (Special case of Fact 3 with K equal to the cone of positive semi-definite 
matrices in the real space of n x n Hermitian matrices, and R(H) = AHA*, P(H) = 
Le, CrHC%). Let A,Cy,k = 1,...,5 be complex n x n matrices which can be simultane- 
ously upper triangularized by similarity. Then there exists a natural correspondence «;, yi of the 
eigenvalues of A, C;,, k = 1,...s. For Hermitian H, let T(H) = AH A* — Dj_,C;, HC. Then the 
following are equivalent: 

|? 


(a) |a;|* — pa aia >0,i=1,...,n. 


(b) For all positive definite G there exists a (unique) positive definite H such that T(H) = G. 
(c) There exists a positive definite H such that T(H) is positive definite. 


10. Gantmacher-Lyapunov [Gan59, Chapter XV] (Special case of Fact 9 with A replaced by A+ I,s = 
2,C,; = A,C, = I, and special case of Fact 7 with K equal to the cone of positive semi-definite 
matrices in the real space of n x n Hermitian matrices and T(H) = AH + HA*). 

Let A € C”””. The following are equivalent: 


(a) Forall positive definite G there exists a (unique) positive definite H such that AH+H A* = G. 
(b) There exists a positive definite H such that AH + H A* is positive definite. 
(c) A is positive stable. 


11. Stein [Ste52](Special case of Fact 9 with A = I,s = 1,C,; = C, and special case of Fact 7 with 
T(H) = H—-CHC%*). 
Let C € C"*”. The following are equivalent: 


(a) There exists a positive definite H such that H — C HC% is positive definite. 
P P 


(b) The spectrum of C is contained in the open unit disk. 


Examples: 


0 
1 
1 > 0 whereas (e;,e2) = 0; so —T is not cross-positive on K and hence not K -resolvent nonnegative. 
Since the eigenvalues of T are —1 and 1, T is not positive stable. This example tells us that the converse of 
Fact 4 is false. It also shows that to the list of equivalent conditions of Fact 3 we cannot add condition (f) 
of Fact 5. 


Let K = (R{)* and take T = : . Then TK = K and so T(intK) D intK. Note that (Te,,e.) = 
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27.1 Combinatorial Structure and Invariants 


The combinatorial structure of a matrix generally refers to the locations of the nonzero entries of a 
matrix, or it might be used to refer to the locations of the zero entries. To study and take advantage of the 
combinatorial structure of a matrix, graphs are used as models. Associated with a matrix are several graphs 
that represent the combinatorial structure ofa matrix in various ways. The type of graph (undirected graph, 
bipartite graph, digraph) used depends on the kind of matrices (symmetric, rectangular, square) being 
studied ([BR91], [Bru92], [BS04]). Conversely, associated with a graph, bipartite graph, or digraph are 
matrices that allow one to consider it as an algebraic object. These matrices — their algebraic properties — 
can often be used to obtain combinatorial information about a graph that is not otherwise obtainable. 
These are two of three general aspects of combinatorial matrix theory. A third aspect concerns intrinsic 
combinatorial properties of matrices viewed simply as an array of numbers. 


Definitions: 


Let A = [a;;] bean m x n matrix. 

A strong combinatorial invariant of A is a quantity or property that does not change when the rows 
and columns of A are permuted, that is, which is shared by all matrices of the form P AQ, where P is a 
permutation matrix of order m and Q is a permutation matrix of order n. 

A less restrictive definition can be considered when A is a square matrix of order n. 

A weak combinatorial invariant is a quantity or property that does not change when the rows and 
columns are simultaneously permuted, that is, which is shared by all matrices of the form P AP T where 
P is a permutation matrix of order n. 

The (0, 1)-matrix obtained from A by replacing each nonzero entry with a 1 is the pattern of A. 
(In those situations where the actual value of the nonzero entries is unimportant, one may replace a matrix 
with its pattern, that is, one may assume that A itself is a (0, 1)-matrix.) 


27-1 


27-2 Handbook of Linear Algebra 


A line of a matrix is a row or column. 

A zero line is a line of all zeros. 

The term rank of a (0, 1)-matrix A is the largest size 9(A) of a collection of 1s of A with no two 1s in 
the same line. 

A cover of A is a collection of lines that contain all the 1s of A. 

A minimum cover is a cover with the smallest number of lines. The number of lines in a minimum line 
cover of A is denoted by c(A). 

A co-cover of A is a collection of 1s of A such that each line of A contains at least one of the 1s. 

A minimum co-cover is a co-cover with the smallest number of 1s. The number of 1s in a minimum 
co-cover is denoted by c*(A). 

The quantity 0*(A) is the largest size of a zero submatrix of A, that is, the maximum ofr + s taken 
over all integers r ands withO <r < mand0 <s <nsuch that A hasanr x s zero (possibly vacuous) 
submatrix. 


Facts: 
The following facts are either elementary or can be found in Chapters 1 and 4 of [BR91]. 


1. These are strong combinatorial invariants: 
(a) The number of rows (respectively, columns) of a matrix. 
(b) The quantity max{r,s} taken over all r x s zero submatrices (0 < 1,5). 
(c) The maximum value ofr + s taken over all r x s zero submatrices (0 < r,s). 
(d) The number of zeros (respectively, nonzeros) in a matrix. 
(e) The number of zero rows (respectively, zero columns) of a matrix. 
(f) The multiset of row sums (respectively, column sums) of a matrix. 
(g) The rank of a matrix. 
(h) The permanent (see Chapter 31) of a matrix. 
(i) The singular values of a matrix. 
2. These are weak combinatorial invariants: 
(a) The largest order of a principal submatrix that is a zero matrix. 
(b) The number of A zeros on the main diagonal of a matrix. 


(c) The maximum value of p + q taken over all p x q zero submatrices that do not meet the main 
diagonal. 


(d) Whether or not for some integer r with 1 < r < n, the matrix A of order n has anr x n—r 
zero submatrix that does not meet the main diagonal of A. 


(e) Whether or not A is a symmetric matrix. 

(f) The trace tr(A) of a matrix A. 

(g) The determinant det A of a matrix A. 

(h) The eigenvalues of a matrix. 

(i) The multiset of elements on the main diagonal of a matrix. 


3. @(A),c(A),0*(A), and c*(A) are all strong combinatorial invariants. 

4. p(A) =c(A). 

5. A matrix A has a co-cover if and only if it does not have any zero lines. If A does not have any zero 
lines, then 9*(A) = c*(A). 

6. If Aisan m x n matrix without zero lines, then @(A) + @*(A) = c(A) +c*(A) = m+n. 
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7. rank(A) < @(A). 
8. Let A be an m x n (0,1)-matrix. Then there are permutation matrices P and Q such that 


Ay. OY LZ 
"Ae (OS 50 
MOP, 5° fe: 0" 
6: -3Es 3070 


where Aj, Az, and A3 are square, possibly vacuous, matrices with only 1s on their main diagonals, 
and (A) is the sum of the orders of A}, Ay, and A3. The rows, respectively columns, of A that are 
in every minimum cover of A are the rows, respectively columns, that meet Aj, respectively Ap. 
These rows and columns together with either the rows that meet A; or the columns that meet A; 
form minimum covers of A. 


Examples: 
1. Let 


> 

II 
oojojcoc O|co = 
oojoo Of} ke 
Re RR Re =| Or 
FF Ol; = Ol] eR 
oO O;rF;OoO O;CoO Fr 
ooro;ooor 
oO OlOo|o Ol] & 


Then @(A) = c(A) = 5 with the five Is in different lines, and rows 1, 2, and 5 and columns 3 and 
4 forming a cover. The matrix is partitioned in the form given in Fact 8. 


27.2 Square Matrices and Strong Combinatorial Invariants 


In this section, we consider the strong combinatorial structure of square matrices. 


Definitions: 


Let A be a (0, 1)-matrix of order n. 

A collection of n nonzero entries in A no two on the same line is a diagonal of A (this term is also 
applied to nonnegative matrices). 

The next definitions are concerned with the existence of certain zero submatrices in A. 

A is partly decomposable provided there exist positive integers p and q with p + q = n such that A 
has a p x q zero submatrix. Equivalently, there are permutation matrices P and Q and an integer k with 
1 <k <n-—1 such that 


B C 


PAQ = 
Q Ogn-k D 


A is a Hall matrix provided there does not exist positive integers p and q with p + q > n such that A 
has a p x q zero submatrix. 

A has total support provided A 4 O and each 1 of A is ona diagonal of A. 

A is fully indecomposable provided it is not partly decomposable. 

A is nearly decomposable provided it is fully indecomposable and each matrix obtained from A by 
replacing a 1 with a 0 is partly decomposable. 
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Facts: 


Unless 


1, 


2: 
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otherwise noted, the following facts can be found in Chapter 4 of [BR91]. 


[BS94] Each of the following properties is equivalent to the matrix A of order n being a Hall matrix: 
(a) p(A) = n, that is, A has a diagonal (Frobenius—Kéonig theorem). 
(b) Forall nonempty subsets L of {1,2,... ,n}, A[{1,2,...,n}, L] has at least |L| nonzero rows. 


(c) For all nonempty subsets K of {1,2,...,n}, A[K,{1,2,...,n}] has at least |K| nonzero 
columns. 


Each of the following properties is equivalent to the matrix A of order n being a fully indecomposable 
matrix: 


(a) p(A) = nand the only minimum line covers are the set of all rows and the set of all columns. 


(b) For all nonempty subsets L of {1,2,... , a}, A[{1,2,...,n}, L] has at least |L| + 1 nonzero 
rows. 


(c) For all nonempty subsets K of {1,2,... ,n}, A[K, {1,2,...,/}] has at least |K| + 1 nonzero 
columns. 


(d) The term rank p(A(i, j)) of the matrix A(i, j) obtained from A by deleting rowi and column 
j equals n — 1 for alli, 7 = 1,2,...,n. 


(e) A"! is a positive matrix. 


(f) The determinant det Ao X of the Hadamard product of A with a matrix X = [x;;] of distinct 
indeterminates over a field F is irreducible in the ring F [{x;j : 1 < i,j < n}]. 


Each of the following properties is equivalent to the matrix A of order n having total support: 
(a) A ¥ O and the term rank p(A(i, j)) equals n — 1 for alli, j = 1,2,...,n with ajj 4 0. 


(b) There are permutation matrices P and Q such that P AQ is a direct sum of fully indecom- 
posable matrices. 


(Dulmage—Mendelsohn Decomposition theorem) If the matrix A of order n has term rank equal to 
n, then there exist permutation matrices P and Q and an integer t > 1 such that 


Ay Ay ++: Aut 
O Ar «+ Ay 
PAQ=]., p : i) es 
O O -. A, 
where Aj, A2,... , Ay are square fully indecomposable matrices. The matrices Aj, A2,... , Ar are 


called the fully indecomposable components of A and they are uniquely determined up to per- 
mutations of their rows and columns. The matrix A has total support if and only if Aj; = O for all 
i and j withi < j; A is fully indecomposable if and only if t = 1. 

(Inductive structure of fully indecomposable matrices) If A is a fully indecomposable matrix of order 
n, then there exist permutation matrices P and Q and an integer k > 2 such that 


B, O ::» O Ey 
E, By -+- O O 
PaQ=]i io: if, 
O O -::+ Bey O 
O O «-:- Ex By 
where Bj, Bo,..., Bg are fully indecomposable and EF), E2,...,£, each contain at least one 


nonzero entry. Conversely, a matrix of such a form is fully indecomposable. 
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6. (Inductive structure of nearly decomposable matrices) If A is a nearly decomposable (0, 1)-matrix, 
then there exist permutation matrices P and Q and an integer p with 1 < p < n— 1 such that 


100 :-:- 0 0 

1 10 :--. 0 0 

0 1 1 ::- 0 0 

Be Bama Wk Se dle pas || Hed 
PAQ= ar Se Ua Ow ; 

0 0 0 -:-- 1 0 

0 0 0 1 ol 

F, A’ 


where A’ is a nearly decomposable matrix of order n — p, the matrix F, has exactly one 1 and this 
1 occur in its first row, and the matrix F) has exactly one 1 and this 1 occurs in its last column. If 
n—p > 2,and the 1 in F isinits column j and the 1 in F) is in its row/, then the (i, 7) entry of A’ is 0. 
7. The number of nonzero entries in a nearly decomposable matrix A of order n > 3 is between 2n 


and 3(n — 1). 
Examples: 
1. Let 
1 0 0 1 0 1 1 0 1 1 0 
A,=]1 0 0], A.=]1 O|},A;= 1/1 1 O], As=]0 1 1 
1 1éi1 1 1 0 0 1 1 0 1 


Then A, is partly decomposable and not a Hall matrix. The matrix A) is a Hall matrix and is partly 
decomposable, but does not have total support. The matrix A; has total support. The matrix Ay, is 
nearly decomposable. 


27.3 Square Matrices and Weak Combinatorial Invariants 


In this section, we restrict our attention to the weak combinatorial structure of square matrices. 


Definitions: 


Let A be a matrix of order n. 

B is permutation similar to A ifthere exists a permutation matrix P suchthat B = P’ AP (= P~!AP). 
A is reducible provided n > 2 and for some integer r with 1 < r < n—1, there exists anr x (n —r) zero 
submatrix which does not meet the main diagonal of A, that is, provided there is a permutation matrix P 


B Cc 
Orn—r D|’ 


A is irreducible provided that A is not reducible. 

A is completely reducible provided there exists an integer k > 2 anda permutation matrix P such that 
PAP™ = A; ® Ap ®--- @ Ax where Aj, Ay,... , Ag are irreducible. 

Ais nearly reducible provided A is irreducible and each matrix obtained from A by replacing a nonzero 
entry with a zero is reducible. 

A Frobenius normal form of A is a block upper triangular matrix with irreducible diagonal blocks that is 
permutation similar to A; the diagonal blocks are called the irreducible components of A. (cf. Fact 27.3.) 


and an integer r with 1 < r < n— 1 such that 


PAP? = 


The following facts can be found in Chapter 3 of [BR91]. 
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Facts: 
1. (Frobenius normal form) There is a permutation matrix P and an integer r > 1 such that 


Ay Ay: At 
On Ape een Age 
PAPT=|]. |, 5 of 
O O =: A; 
where Aj, A2,... , A; are square irreducible matrices. The matrices A), A2,..., A; are the irre- 


ducible components of A and they are uniquely determined up to simultaneous permutations of 
their rows and columns. 

2. There exists a permutation matrix Q such that AQ is irreducible if and only if A has at least one 
nonzero element in each line. 

3. If A does not have any zeros on its main diagonal, then A is irreducible if and only if A is fully 
indecomposable. The matrix A is fully indecomposable if and only if there is a permutation matrix 
Q such that AQ has no zeros on its main diagonal and AQ is irreducible. 

4. (Inductive structure of irreducible matrices) Let A be an irreducible matrix of order n > 2. Then 
there exists a permutation matrix P and an integer m > 2 such that 


A; O =: O E, 
E, Ay °: O O 
PAPP=}> io th, 
O O ::: Any O 
O O .:-: Em Am 
where Aj, Ao,... , Aj are irreducible and Ej, E,... , Ej, each have at least one nonzero entry. 


5. (Inductive structure of nearly reducible matrices) If A is a nearly reducible (0, 1)-matrix, then there 
exist permutation matrix P and an integer m with 1 < m < n— 1 such that 


0 0 0 :-- 0 0 
100 :-. 0 0 
Oe A cae SBOE 
PAPI Sa Se 
0 0 0 1 0 
F, A’ 


where A’ is a nearly reducible matrix of order m, the matrix F, has exactly one 1 and it occurs in 
the first row and column j of F, with 1 < j < m, and the matrix F) has exactly one 1 and it occurs 
in the last column and row i of F, where 1 < i < m. The element in position (i, j) of A’ is 0. 

6. The number of nonzero entries in a nearly reducible matrix of order n > 2 is between n and 


2(n — 1) 
Examples: 
1. Let 
1 0 0 1 11 1 0 0 0 1 0 
Av=|1 1 1),4,=]1 0 1|,A4;=/0 1 1|,4,=|0 0 1). 
1 11 1 01 0 1 1 1 0 0 


Then A, is reducible but not completely reducible, and A, is irreducible. (Both A; and A) are 
partly decomposable.) The matrix A; is completely reducible. The matrix A, is nearly reducible. 
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27.4 The Class A(R,S) of (0,1)-Matrices 


In the next definition, we introduce one of the most important and widely studied classes of (0, 1)-matrices 
(see Chapter 6 of [Rys63] and [Bru80]). 


Definitions: 

Let A = [a;;] bean m x n matrix. 
The row sum vector of Ais R = (r1,1r2,... 57m), Wherer; = Dot aij, (i = 1,2,... ,n). 
The column sum vector of A is S = (s),52,... ,5,), where sj = eeu aij, (j =1,2,...,n). 
A real vector (c1,C2,... ,C,) is monotone provided c, > c2 > +--+ > Cy. 


The class of all m x n (0, 1)-matrices with row sum vector R and column sum vector S is denoted by 
A(R, S). 

The class A(R, S) isa monotone class provided R and S are both monotone vectors. 

An interchange is a transformation on a (0, 1)-matrix that replaces a submatrix equal to the identity 


matrix I, by the submatrix 
0 1 
L = 
or vice versa. 


If (A) is any real numerical quantity associated with a matrix A, then the extreme values of 6 are 
6(R, S) and 6(R, S), defined by 


6(R, S) = max{6(A): A € A(R, S)} and 6(R, S) = min{@(A): A € A(R, S)}. 


Let T = [ty] be the (m+ 1) x (n + 1) matrix defined by 


i=k+1 


1 m 
ty =kKl- Sos; + So rj, (kK =0,1,...,m51=0,1,...,n). 
j=l 


The matrix T is the structure matrix of A(R, S). 


Facts: 


The following facts can be found in Chapter 6 of [Rys63], [Bru80], Chapter 6 of [BR91], and Chapters 3 
and 4 of [Bru06]. 


1. Aclass A(R, S) can be transformed into a monotone class by row and column permutations. 

2. Let U = (uj, u2,...,U,) and V = (v1,¥2,...,V,) be monotone, nonnegative integral vectors. 
U X V if and only if V* < U*, and U** = U or U extended with 0s. 

3. (Gale—Ryser theorem) A(R, S) is nonempty if and only if S = R*. 

4. Let the monotone class A(R,S) be nonempty, and let A be a matrix in A(R,S). Let K = 
{1,2,...,k}andL = {1,2,... ,1}. Then t equals the number of 0s in the submatrix A[K, L] plus 
the number of 1s in the submatrix A(K, L); in particular, we have t > 0. 

5. (Ford—Fulkerson theorem) The monotone class A(R, S) is nonempty if and only if its structure 
matrix T is a nonnegative matrix. 

6. If Ais in A(R, S) and B results from A by an interchange, then B is in A(R, S). Each matrix in 
A(R, S) can be transformed to every other matrix in A(R, S) by a sequence of interchanges. 

7. The maximum and minimum term rank of a nonempty monotone class A(R, S) satisfy: 


A(R, S) = min{h) +k +1;k =0,1,...,m,1=0,1,... ,n}, 
A(R, S) = min{k +1: be > ti,k =0,1,...,m,1=0,1,...,n}, 
where 


ba = min{h, 145, + trig +A - 1) - fi}, 
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the minimum being taken over all integers 7), 72, j1, jy such that0 <7) < k < k+i. < mand 
O<fj<l<ltp<n. 

8. Let tr(A) denote the trace of a matrix A. The maximum and minimum trace of a nonempty 
monotone class A(R, S) satisfy: 


tr(R, S) = min{t + max{k,]}:0<k < m,0 <I <n}, 


tr(R, S) = max{min{k,]} —  :0 < k < m,0 <1 <n}. 


9. Let k and n be integers with 0 < k < n, and let A(n,k) denote the class A(R, S), where R = S = 


(k,k,...,k) (nk’s). Let 6(n, k) and 5(n, k) denote the minimum and maximum rank, respectively, 
of matrices in A(n, k). 

0, ifk = 0, 
CCH an eae 


3,ifk =2andn=4, 
n, otherwise. 


(b) b(n,k) =Dn,n—k)ifl<k<n-1, 

(c) b(n,k) > [n/k], (1 < k < n— 1), with equality if and only if k divides n. 
(d) d(n,k) < [n/k] +k, 1 sk <n). 

(e) b(n,2) = n/2 if n is even, and (n + 3)/2 if n is odd. 

(f) b(n, 3) = n/3 if 3 divides n and |n/3| + 3 otherwise. 


Additional properties of A(R, S) can be found in [Bru80] and in Chapters 3 and 4 of [Bru06]. 


Examples: 
1. Let R = (7,3,2,2,1,1) and S = (5,5,3,1,1,1). Then R* = (6,4,2,1,1,1,1). Sincee5+5+3> 


6+4+2,S & R* and, by Fact 3, A(R, S) =. 
2. Let R= S = (2,2, 2, 2,2). Then the matrices 


1 10 0 0 100 0 1 
0 1 1 £0 0 0 1 1 £0 0 
A=1!10 0 1 1 Of andB=1]0 1 0 1 0 
0001 1 0 01 1 +0 
100 0 1 100 0 1 


1 10 0 0 100 0 1 100 0 1 
0 1 1 £0 0 0 1 1 0 0 0 1 1 °0 0 
0 01 1 0);>4j0 021 1 0; >)]0 10 1 «0 
000141 0 10 1 0 0 0 1 1 +0 
1 00 0 1000 1 100 0 1 


27.5 The Class T(R) of Tournament Matrices 


In the next definition, we introduce another important class of (0, 1)-matrices. 


Definitions: 


A (0,1)-matrix A = [ajj] of order n is a tournament matrix provided a;; = 0, (1 < i < n) and 
aij +aji =1,(1 <i < j <n), that is, provided A+ Al =J,—Iy. 
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The digraph of a tournament matrix is called a tournament. 

Thinking of n teams pj, p2,... » Pn playingina round-robin tournament, we have that a;; = | signifies 
that team p; beats team pj. 

The row sum vector R is also called the score vector of the tournament (matrix). 

A transitive tournament matrix is one for which a;; = aj, = 1 implies aj, = 1. 

The class of all tournament matrices with score vector R is denoted by T(R). 

A 6-interchange is a transformation on a tournament matrix that replaces a principal submatrix of 
order 3 equal to 


or vice versa. 
The following facts can be found in Chapters 2 and 5 of [Bru06]. 


Facts: 


1. The row sum vector R = (r1,1r2,...57,) and column sum vector S = (s1,52,... 55,) of a tourna- 
ment matrix of order n satisfy r; + 5; = n — 1, (1 <i <n); in particular, the column sum vector 
is determined by the row sum vector. 

2. If Ais a tournament matrix and P is a permutation matrix, then P AP T is a tournament matrix. 
Thus, one may assume without loss of generality that R is nondecreasing, thatis,r) < rz < +--+ < Th; 
so that the teams are ordered from worst to best. 

3. (Landaw’s theorem) If R = (11,12,...51n) is a nondecreasing, nonnegative integral vector, then 
T (R) is nonempty if and only if 


s k 
See a (l<k<n) 
i=1 


with equality when k = n. (A binomial coefficient () is Oifk <s.) 
4, Let R = (rj,12,... 1») be a nondecreasing nonnegative integral vector. The following are equiva- 
lent: 


(a) There exists an irreducible matrix in T(R). 
(b) Z(R) is nonempty and every matrix in T (R) is irreducible. 
(c) Shiri = (§),(. < k <n) with equality if and only if k = n. 


5. If Aisin J(R) and B results from A by a 5-interchange, then B is in J (R). Each matrix in T(R) 
can be transformed to every other matrix in J (R) by a sequence of 5-interchanges. 

6. The rank, and so term rank, of a tournament matrix of order n is at least n — 1. 

7. (Strengthened Landau inequalities) Let R = (11,12,... Tn) beanondecreasing nonnegative integral 
vector. Then J (R) is nonempty if and only if 


eae soe-n45 (UI) (S41, 2,0... 5n))5 


iel iel 


with equality if I = {1,2,...,n}. 

8. Let R = (r1,12,...51,) be a nondecreasing, nonnegative integral vector such that T(R) is 
nonempty. Then there exists a tournament matrix A such that the principal submatrices made 
out of the even-indexed and odd-indexed, respectively, rows and columns are transitive tourna- 
ment matrices, that is, a matrix A in J(R) such that A[{1,3,5,...}] and A[{2,4,6,...}] are 
transitive tournament matrices. 
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Examples: 


1. The following are tournament matrices: 


Aj = and A; = 


re OO 
oo Ff 
ore 
—_- — — © 
ere OC 


re OO oO 
ooo °f 


The matrix A) is a transitive tournament matrix. 
2. Let R = (2,2,2,2,3,4). A tournament matrix in J (R) satisfying Fact 7 is 


= - OK © 
a a OS oS 
CoOoroorrF 
> a) 
FP OoOrdoeoedced 
oo or OC 8 


since the two submatrices 


1 0 
A[I[1, 3, 5}] = 1] and A[{2,4,6}] = |0 
0 1 


oro 
oS: 
Ke Or 
oOo oO 


are transitive tournament matrices. 


27.6 Convex Polytopes of Doubly Stochastic Matrices 


Doubly stochastic matrices (see Chapter 9.4) are widely studied because of their connection with probability 
theory, as every doubly stochastic matrix is the transition matrix of a Markov chain. The reader is referred 
to Chapter 28 for graph terminology. 


Definitions: 


Since a convex combination c A + (1 — c)B of two doubly stochastic matrices A and B, where 0 <c <1, 
is doubly stochastic, the set Q, of doubly stochastic matrices of order n is a convex polytope in R”. 
Q, is also called the assignment polytope because of its appearance in the classical assignment 
problem. 

If A is a (0, 1)-matrix of order n, then F(A) is the convex polytope of all doubly stochastic matrices 
whose patterns P satisfy P < A (entrywise), that is, that have 0s at least wherever A has Os. 

The dimension of a convex polytope P is the smallest dimension of an affine space containing it and is 
denoted by dim P. 

The graph G(P) of a convex polytope P has the extreme points of P as its vertices and the pairs of 
extreme points of one-dimensional faces of P as its edges. 

The chromatic index of a graph is the smallest integer t such that the edges can be partitioned into sets 
E,, E2,... , E; such that no two edges in the same E; meet. 

A scaling of a matrix A is a matrix of the form D,; AD), where D, and D) are diagonal matrices with 
positive diagonal entries. 

If D| = Dy», then the scaling D,; AD) is a symmetric scaling. 

Let A = [a;;] have order n. A diagonal product of A is the product of the entries on a diagonal of A, 
that is, a; j,42j, +++ @nj, Where jy, jx... » jn is a permutation of {1,2,... , n}. 

The following facts can be found in Chapter 9 of [Bru06]. 
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Facts: 
1. 


2 
3% 


(Birkhoff’s theorem) The extreme points of Q,, are the permutation matrices of order n. Thus, each 
doubly stochastic matrix is a convex combination of permutation matrices. 

The patterns of matrices in Q,, are precisely the (0, 1)-matrices of order n with total support. 

The faces of Q,, are the sets F(A), where A is a (0, 1)-matrix of order n with total support. Q, is a 
face of itself with Q,, = F(J,,). The dimension of F(A) satisfies 


dim F(A) =t—2n+k, 


where t¢ is the number of 1s of A and k is the number of fully indecomposable components of A. 
The number of extreme points of F(A) (this number is the permanent per(A) of A), is at least 
t—2n+k+1.If A is fully indecomposable and dim F(A) = d, then F(A) has at most 24-14] 
extreme points. In general, F(A) has at most 2¢ extreme points. 


. The graph G(Q,,) has the following properties: 


(a) The number of vertices of G(Q,,) is n!. 
(b) The degree of each vertex of G(Q,,) is 


(c) G(Q,,) is connected and its diameter equals 1 if n = 1,2, and 3, and equals 2 ifn > 4. 
(d) G(Q,) has a Hamilton cycle. 
(e) The chromatic index of G(Q,,) equals dy. 


. (Hardy, Littlewood, Polya theorem) Let U = (uy, uU2,...,Un) and V = (11, V2,... ,V_) be mono- 


tone, nonnegative integral vectors. Then U < V if and only if there is a doubly stochastic matrix 
A such that U = VA. 


. The set Y,, of symmetric doubly stochastic matrices of order n is a subpolytope of Q,, whose extreme 


points are those matrices A such that there is a permutation matrix P for which PAP? is a direct 


sum of matrices each of which is either the identity matrix I, of order 1, the matrix ,oran 
odd order matrix of the type: 

0 1/2 0 0 0 1/2 

1/72 0 1/2 0 0 0 

Or 4. Oe TP 0 0 

0 oO 1/2 0 0 0 

0 0 0 0 see 0 1/2 

12 90 0 OO wee Te 20 


. Let A be a nonnegative matrix. Then there is a scaling B = D, AD, of A that is doubly stochastic 


if and only if A has total support. If A is fully indecomposable, then the doubly stochastic matrix 
B is unique and the diagonal matrices D, and D) are unique up to reciprocal scalar factors. 


. Let A be a nonnegative symmetric matrix with no zero lines. Then there is a symmetric scaling 


B = DADsuch that B is doubly stochastic if and only if A has total support. If A is fully indecom- 
posable, then the doubly stochastic matrix B and the diagonal matrix D are unique. 


. Distinct doubly stochastic matrices of order n do not have proportional diagonal products; that is, 


if A = [a;;] and B = [b;;] are doubly stochastic matrices of order n with A ¥ B, there does not 
exist a constant c such that 41 ;,42), +++ nj, = €b1),b2;,---bnj, for all permutations j,, j2,... , jn 
of {1,2,... ,n}. 
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The subpolytopes of Q,, consisting of (i) the convex combinations of the n! — 1 nonidentity permutation 


matrices of order n and (ii) the permutation matrices corresponding to the even permutations of order n 
have been studied (see [Bru06]). 


Polytopes of matrices more general than Q, have also been studied, for instance, the nonnegative 
generalizations of A(R, S) consisting of all nonnegative matrices with a given row sum vector R and a 
given column sum vector S (see Chapter 8 of [Bru06]). 


Examples: 


1. Q) consists of all matrices of the form 


fe h (0<a<1). 


l—-a a 
All permutation matrices are doubly stochastic. 
2. The matrix 
1/2 1/4 1/4 
1/6 1/3 1/2 
1/3 5/12 1/4 


is a doubly stochastic matrix of order 3. 


3. If 
110 
A=|0 1 1], 
111 
then 
12> D/d; 0 
GO. 18s Jy 
ie 0: At 
isin F(A). 
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The first two sections of this chapter “Matrices and Graphs” give a short introduction to graph theory. 
Unfortunately much graph theoretic terminology is not standard, so we had to choose. We allow, for 
example, graphs to have multiple edges and loops, and call a graph simple if it has none of these. On the 
other hand, we assume that graphs are finite. 

For all nontrivial facts, references are given, sometimes to the original source, but often to text books 
or survey papers. A recent global reference for this chapter is [BW04]. (This book was not available to the 
author when this chapter was written, so it is not referred to in the text below.) 


28.1 Graphs: Basic Notions 


Definitions: 


A graph G = (V, E) consists of a finite set V = {v1,... , v,} of vertices and a finite multiset E of edges, 
where each edge is a pair {v;, v;} of vertices (not necessarily distinct). If vj = v;, the edge is called a loop. 
A vertex v; of an edge is called an endpoint of the edge. 

The order of graph G is the number of vertices of G. 

A simple graph is a graph with no loops where each edge has multiplicity at most one. 

Two graphs (V,E) and (V’, E’) are isomorphic whenever there exist bijections ¢ : V — V’ and 
w:E — E',such that v € V isan endpoint of e € E if and only if }(v) is an endpoint of y(e). 

A walk of length @ in a graph is an alternating sequence (¥j,, €),5 Vi,» €i,s-- -» Ci,» Vi,) of vertices and edges 
(not necessarily distinct), such that v;, , and v;, are endpoints of e;, for j = 1,... ,€. 

A path of length @ in a graph is a walk of length € with all vertices distinct. 

A cycle of length ¢ in a graph is a walk (V;,, €i,, Vi,» @i>- ++ »@i» Vi.) With Vi, = vi,,€ A 0, and vj,,... 5 Vi, 
all distinct. 

A Hamilton cycle in a graph is a cycle that includes all vertices. 

A graph (V, E) is connected if V # @ and there exists a walk between any two distinct vertices of V. 

The distance between two vertices v; and v; of a graph G (denoted by dg(v;,v;) or d(vj,v;)) is the 
length of a shortest path between v; and vj. (d(vj,vj) = Oifi = j, and d(v;,v;) is infinite if there is no 
path between v; and vj.) 

The diameter of a connected graph G is the largest distance that occurs between two vertices of G. 
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A tree is a connected graph with no cycles. 

A forest is a graph with no cycles. 

A graph (V’, E’) is a subgraph of a graph (V, E) if V’ C Vand E’ C E. If E’ contains all edges from 
E with endpoints in V’, (V’, E’) is an induced subgraph of (V, E). 

A spanning subgraph of a connected graph (V, E) is a subgraph (V’, E’) with V’ = V, which is 
connected. 

A spanning tree of a connected graph (V, E) is a spanning subgraph, which is a tree. 

A connected component of a graph (V, E) is an induced subgraph (V’, E’), which is connected and 
such that there exists no edge in E with one endpoint in V’ and one outside V’. A connected component 
with one vertex and no edge is called an isolated vertex. 

Two graphs (V, E) and (V’, E’) are disjoint if V and V’ are disjoint sets. 

Two vertices u and v are adjacent if there exists an edge with endpoints u and v. A vertex adjacent to v 
is called a neighbor of v. 

The degree or valency of a vertex v of a graph G (denoted by 5g(v) or 5(v)) is the number of times that 
v occurs as an endpoint of an edge (that is, the number of edges containing v, where loops count as 2). 

A graph (V, E) is bipartite if the vertex set V admits a partition into two parts, such that no edge of E 
has both endpoints in one part (thus, there are no loops). More information on bipartite graphs is given 
in Chapter 30. 

A simple graph (V, E) is complete if E consists of all unordered pairs from V. The (isomorphism class 
of the) complete graph on n vertices is denoted by K,. 

A graph (V, E) is empty if E = %. Ifalso V = Q, it is called the null graph. 

A bipartite simple graph (V, E) with nonempty parts V| and V) is complete bipartite if E consists of 
all unordered pairs from V with one vertex in V; and one in V3. The (isomorphism class of the) complete 
bipartite graph is denoted by Ky,,n,, where n) = |Vi| and nz = | V4]. 

The (isomorphism class of the) simple graph that consists only of vertices and edges of a path of length 
£ is called the path of length @, and denoted by P,+;. 

The (isomorphism class of the) simple graph that consists only of vertices and edges of a cycle of length 
£ is called the cycle of length ¢, and denoted by Cy. 

The complement of a simple graph G = (V, E) is the simple graph G = (V, E), where E consists of 
all unordered pairs from V that are not in E. 

The union G U G’ of two graphs G = (V, E) and G’ = (V’, E’) is the graph with vertex set V U V’, 
and edge (multi)set E U E’. 

The intersection G M G’ of two graphs G = (V,E) and G’ = (V’, E’) is the graph with vertex set 
VV’, and edge (multi)set EN E’. 

The join G + G’ of two disjoint graphs G = (V, E) and G’ = (V’, E’) is the union of G U G’ and the 
complete bipartite graph with vertex set V U V’ and partition {V, V’}. 

The (strong) product G - G’ of two simple graphs G = (V, E) and G’ = (V’, E’) is the simple graph 
with vertex set V x V’, where two distinct vertices are adjacent whenever in both coordinate places the 
vertices are adjacent or equal in the corresponding graph. The strong product of £ copies of a graph G is 
denoted by G*. 


Facts: 


The facts below are elementary results that can be found in almost every introduction to graph theory, 
such as [Har69] or [Wes01]. 


1. For any graph, the sum of its degrees equals twice the number of edges; therefore, the number of 
vertices with odd degree is even. 

2. For any simple graph, at least two vertices have the same degree. 

3. A graph G is bipartite if and only if G has no cycles of odd length. 

4. A tree with n vertices has n — 1 edges. 


Matrices and Graphs 28-3 


Gy G3 


FIGURE 28.1 Three graphs. (Vertices are represented by points and an edge is represented by a line segment between 
the endpoints, or a loop.) 


5. A graph is a tree if and only if there is a unique path between any two vertices. 
6. A graph G is connected if and only if G cannot be expressed as the union of two or more mutually 
disjoint connected graphs. 


Examples: 


1. Consider the complete bipartite graph K33 = (V, U Vj, E) with parts V| = {vj,v2,v3} and 
Vo = {V4 V5, Vo}. Then 


V1s{V1, V5}, V55 {55 V2}, Vas {V2 Vo}, Vor (V6, V3}, V3 
is a path of length 4 between v, and v3, 

V15{V1, V5}, V55 (V5) V2}, Vas {V2 V5}, V55 (V5, V3}, V3 
is a walk of length 4, which is not a path, and 

Vis {V5 V5}, V55 {55 V2}, Vas {V2 Vo}, Vos {Yor Vi}> V1 


is a cycle of length 4. 

Graphs G, and G) in Figure 28.1 are simple, but G3 is not. 

Graphs G, and G; are bipartite, but G; is not. 

Graph G;, isa tree. Its diameter equals 4. 

Graph G) is not connected; it is the union of two disjoint graphs, a path P3 and a cycle C4. The 


SU ecb. 


complement G) is connected and can be expressed as the join of P; and C4. 

6. Graph G3; contains three kinds of cycles, cycles of length 3 (corresponding to the two triangles), 
cycles of length 2 (corresponding to the pair of multiple edges), and one of length 1 (corresponding 
to the loop). 


28.2 Special Graphs 


A graph G is regular (or k-regular) if every vertex of G has the same degree (equal to k). 
A graph G is walk-regular if for every vertex v the number of walks from v to v of length ¢ depends 
only on é (not on v). 
A simple graph G is strongly regular with parameters (n, k, A, 2) whenever G has n vertices and 
* Gis k-regular with 1 <k <n—2. 
* Every two adjacent vertices of G have exactly common neighbors. 
* Every two distinct nonadjacent vertices of G have exactly « common neighbors. 
An embedding of a graph in R” consists of a representation of the vertices by distinct points in R", 
and a representation of the edges by curve segments between the endpoints, such that a curve segment 
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intersects another segment or itself only in an endpoint. (A curve segment between x and y is the range of 
a continuous map @¢ from [0, 1] to R” with @(0) = xand ¢(1) = y.) 

A graph is planar if it admits an embedding in R’. 

A graph is outerplanar if it admits an embedding in R’, such that the vertices are represented by points 
on the unit circle, and the representations of the edges are contained in the unit disc. 

A graph G is linklessly embeddable if it admits an embedding in R’, such that no two disjoint cycles 
of G are linked. (Two disjoint Jordan curves in R® are linked if there is no topological 2-sphere in R° 
separating them.) 

Deletion of an edge e from a graph G = (V, E) is the operation that deletes e from E and results in 
the subgraph G — e = (V,E \ {e}) of G. 

Deletion of a vertex v from a graph G = (V, E) is the operation that deletes v from V and all edges 
with endpoint v from E. The resulting subgraph of G is denoted by G — v. 

Contraction of an edge e of a graph (V, E) is the operation that merges the endpoints of e in V, and 
deletes e from E. 

A minor ofa graph G is any graph that can be obtained from G by a sequence of edge deletions, vertex 
deletions, and contractions. 

Let G bea simple graph. The line graph L(G) of G has the edges of G as vertices, and vertices of L(G) 
are adjacent if the corresponding edges of G have an endpoint in common. 

The cocktail party graph C P(a) is the graph obtained by deleting a disjoint edges from the complete 
graph K»,. (Note that C P (0) is the null graph.) 

Let G be a simple graph with vertex set {v),...,v,}, and let a),... ,a, be nonnegative integers. The 
generalized line graph L(G; a),...,a,) consists of the disjoint union of the line graph L(G) and the 
cocktail party graphs C P(a,),... ,C P(a,), together with all edges joining a vertex {v;,v;} of L(G) with 
each vertex of C P(a;) and C P(a;). 


Facts: 


If no reference is given, the fact is trivial or a classical result that can be found in almost every introduction 
to graph theory, such as [Har69] or [Wes01]. 


1. [God93, p. 81] A strongly regular graph is walk-regular. 

2. A walk-regular graph is regular. 

3. The complement of a strongly regular graph with parameters (n, k, A, jz) is strongly regular with 
parameters (n,n —k—1,n—2k+m—2,n—2k+A). 

4. Every graph can be embedded in R’. 

5. [RS04] (Robertson, Seymour) For every graph property P that is closed under taking minors, there 
exists a finite list of graphs such that a graph G has property P if and only if no graph from the list 
is a minor of G. 

6. The graph properties: planar, outerplanar, and linklessly embeddable are closed undertaking 
minors. 

7. (Kuratowski, Wagner) A graph G is planar ifand only ifno minor of G is isomorphic to Ks or K33. 

[CRS04, p. 8] A regular generalized line graph is a line graph or a cocktail party graph. 

9. (Whitney) The line graphs of two connected nonisomorphic graphs G and G’ are nonisomorphic, 
unless {G, G’} = {K3, K13}. 


00 


Examples: 


1. Graph G; of Figure 28.1 is regular of degree 3. 

2. The complete graph K,, is walk-regular and regular of degree n — 1. 

3. The complete bipartite graph K;, is regular of degree k, walk-regular and strongly regular with 
parameters (2k, k, 0, k). 

4, Examples of outerplanar graphs are all trees, C,,, and Ps. 

5, Examples of graphs that are planar, but not outerplanar are: Ky, C P(3), Cg, and K2_2 for n > 5. 
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6. Examples of graphs that are not planar, but linklessly embeddable 
are: Ks, and K3,,-3 forn > 6. 

7. The Petersen graph (Figure 28.2) and K, forn > 6arenotlinklessly 
embeddable. 

8. Thecomplete graph K; can be obtained from the Petersen graph by 
contraction with respect to five mutually disjoint edges. Therefore, 
K; is a minor of the Petersen graph. 

9. The cycle Co is a subgraph of the Petersen graph and, therefore, the 
Petersen graph has every cycle Cy with £ < 9 as a minor. 

10. Figure 28.3 gives a simple graph G, the line graph L(G), and 
the generalized line graph L(G; 2, 1,0, 0,0) (the vertices of G 
are ordered from left to right). 

11. Forn > 4andk > 2 the line graphs L(K,,) and L(K;,) and their complements are strongly regular. 
The complement of L(K5) is the Petersen graph. 


FIGURE 28.2 The Petersen graph. 


28.3 The Adjacency Matrix and Its Eigenvalues 


Definitions: 


The adjacency matrix Ag ofa graph G with vertex set {v),... , v,} is the symmetric n x n matrix, whose 
(i, j th entry is equal to the number of edges between v; and v;. 

The eigenvalues of a graph G are the eigenvalues of its adjacency matrix. 

The spectrum o(G) of a graph G is the multiset of eigenvalues (that is, the eigenvalues with their 
multiplicities). 

Two graphs are cospectral whenever they have the same spectrum. 

A graph G is determined by its spectrum if every graph cospectral with G is isomorphic to G. 

The characteristic polynomial pg(x) of a graph G is the characteristic polynomial of its adjacency 
matrix Ag, that is, pg(x) = det(xI — Ag). 

A Hoffman polynomial of a graph G is a polynomial h(x) of minimum degree such that h(Ag) = J. 

The main angles of a graph G are the cosines of the angles between the eigenspaces of Ag and the 
all-ones vector 1. 


Facts: 


If no reference is given, the fact is trivial or a standard result in algebraic graph theory that can be found 
in the classical books [Big74] and [CDS80]. 
1. If Ag is the adjacency matrix of a simple graph G, then J — I — Ag is the adjacency matrix of the 
complement of G. 
2. If Ag and Aq are adjacency matrices of simple graphs G and G’, respectively, then ((Ag + I) ® 
(Ag + 1)) — I is the adjacency matrix of the strong product G - G’. 
3. Isomorphic graphs are cospectral. 


> Ot oi 


L(G;2,1,0,0,0) 


FIGURE 28.3 A graph with its line graph and a generalized line graph. 
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Let G be a graph with vertex set {v),... ,v,} and adjacency matrix Ag. The number of walks of 
length ¢ from v; to vj equals (Ag)ij> ie, the i, j-entry of Ag. 

The eigenvalues of a graph are real numbers. 

The adjacency matrix of a graph is diagonalizable. 

IfA, >... > A, are the eigenvalues ofa graph G, then |A;| < 41. IfA, = A2, then G is disconnected. 
If A; = —A, and G is not empty, then at least one connected component of G is nonempty and 
bipartite. 

[CDS80, p. 87] IfA; >... > An are the eigenvalues of a graph G, then G is bipartite if and only if 
Ai = —An4i_i fori = 1,... ,n. For more information on bipartite graphs see Chapter 30. 

If G is a simple k-regular graph, then the largest eigenvalue of G equals k, and the multiplicity of 
k equals the number of connected components of G. 


. [CDS80, p. 94] If A, > ... > A, are the eigenvalues of a simple graph G with n vertices and m 


edges, then >; A? = 2m < ndj. Equality holds if and only if G is regular. 
[CDS80, p. 95] A simple graph G has a Hoffman polynomial if and only if G is regular and 
connected. 


[CRS97, p. 99] Suppose G is a simple graph with n vertices, r distinct eigenvalues v,,... , v,, and 
main angles 6),... , 8. Then the complement G of G has characteristic polynomial 
pox) = (-1)"po(—x — 1) (: nS. Bi/(x+1 2) 
i=l 
[CDS80, p. 103], [God93, p. 179] A connected simple regular graph is strongly regular if and only if 


it has exactly three distinct eigenvalues. The eigenvalues (v) > v2 > v3) and parameters (n, k, A, 1) 
are related by v; = k and 


1 
vada = 5 (A- wt J(u = 2)? + 4k = 1) « 


[BR91, p. 150], [God93, p. 180] The multiplicities of the eigenvalues v,, v2, and v3 of a connected 
strongly regular graph with parameters (n, k, A, w) are 1 and 


:( (n —1)(u — A) — 2k 
n—-1+ 
2 J(u — 2? + 4k = 1) 


[GRO1, p. 190] A regular simple graph with at most four distinct eigenvalues is walk-regular. 


(respectively). 


[CRS97, p. 79] Cospectral walk-regular simple graphs have the same main angles. 

[Sch73] Almost all trees are cospectral with another tree. 

[DH03] The number of nonisomorphic simple graphs on n vertices, not determined by the spec- 
trum, is asymptotically bounded from below by n?g,,_1( x —o(1)), where g,_ denotes the number 
of nonisomorphic simple graphs on n — 1 vertices. 

[DH03] The complete graph, the cycle, the path, the regular complete bipartite graph, and their 
complements are determined by their spectrum. 

[DH03] Suppose G is a regular connected simple graph on n vertices, which is determined by its 
spectrum. Then also the complement G of G is determined by its spectrum, and if n + 1 is nota 
square, also the line graph L(G) of G is determined by its spectrum. 

[CRS04, p. 7] A simple graph G is a generalized line graph if and only if the adjacency matrix Ag 
can be expressed as Ag = C'C — 21, where C is an integral matrix with exactly two nonzero 


entries in each column. (It follows that the nonzero entries are +1.) 

[CRS04, p. 7] A generalized line graph has smallest eigenvalue at least —2. 

[CRS04, p. 85] A connected simple graph with more than 36 vertices and smallest eigenvalue at 
least —2 is a generalized line graph. 

[CRS04, p. 90] There are precisely 187 connected regular simple graphs with smallest eigenvalue 
at least —2 that are not a line graph or a cocktail party graph. Each of these graphs has smallest 
eigenvalue equal to —2, at most 28 vertices, and degree at most 16. 
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0101 0 0010 0 
1000 1 0010 0 
000 0 0 1101 41 
1000 1 0010 0 
0101 0 0010 0 


FIGURE 28.4 Two cospectral graphs with their adjacency matrices. 


Examples: 


1. Figure 28.4 gives a pair of nonisomorphic bipartite graphs with their adjacency matrices. Both 
matrices have spectrum {2, 0°, —2} (exponents indicate multiplicities), so the graphs are cospectral. 

2. The main angles of the two graphs of Figure 28.4 (with the given ordering of the eigenvalues) are 
2/15; 1//5, 0 and 3//10, 0, 1/10, respectively. 

3. The spectrum of Ky,.n, is {,/it2, 0"~-?, —/mny}. 

4. By Fact 14, the multiplicities of the eigenvalues of any strongly regular graph with parameters 
(n, k, 1, 1) would be nonintegral, so no such graph can exist (this result is known as the Friendship 
theorem). 

5. The Petersen graph has spectrum {3, 1°, —2*} and Hoffman polynomial (x — 1)(x + 2). It is one 
of the 187 connected regular graphs with least eigenvalue —2, which is neither a line graph nor a 


cocktail party graph. 
6. The eigenvalues of the path P,, are 2 cos aut (i=1,...,n). 
7. The eigenvalues of the cycle C,, are 2 cos aim (i= 1,...,7). 


28.4 Other Matrix Representations 


Definitions: 


Let G be a simple graph with adjacency matrix Ag. Suppose D is the diagonal matrix with the degrees 
of G on the diagonal (with the same vertex ordering as in Ag). Then Lg = D — Ag is the Laplacian 
matrix of G (often abbreviated to the Laplacian, and also known as admittance matrix), and the matrix 
|Lg| = D+ Ag is (sometimes) called the signless Laplacian matrix. 

The Laplacian eigenvalues of a simple graph G are the eigenvalues of the Laplacian matrix Lg. 

If uy) < fy <... < mt, are the Laplacian eigenvalues of G, then {12 is called the algebraic connectivity 
of G. (See section 28.6 below.) 

Let G be simple graph with vertex set {v,,...,v,}. A symmetric real matrix M = [mj] is called a 
generalized Laplacian of G, whenever mj; < 0 if v; and vj are adjacent, and m;; = 0 if v; and v; are 
nonadjacent and distinct (nothing is required for the diagonal entries of M). 

Let G bea graph without loops with vertex set {v1,... , v,} and edge set {e),... , €m}. The (vertex-edge) 
incidence matrix of G is the n x m matrix Ng defined by (Ng); = 1 if vertex v; is an endpoint of edge 
e; and (Ng )ij = 0 otherwise. 
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An oriented (vertex-edge) incidence matrix of G is a matrix NG obtained from Ng by replacing a 1 in 
each column by a —1, and thereby orienting each edge of G. 

If Ag is the adjacency matrix of a simple graph G, then Sg = J — I — 2.Ag is the Seidel matrix of G. 

Let G bea simple graph with Seidel matrix Sg, and let I’ be a diagonal matrix with +1 on the diagonal. 
Then the simple graph G’ with Seidel matrix Sg, = I'SgI’ is switching equivalent to G. The graph 
operation that changes G into G’ is called Seidel switching. 


Facts: 


In all facts below, G is a simple graph. If no reference is given, the fact is trivial or a classical result that can 
be found in [BR91]. 


1. Let G be a simple graph. The Laplacian matrix Lg and the signless Laplacian |L | are positive 

semidefinite. 

2. The nullity of Lg is equal to the number of connected components of G. 

The nullity of |LG| is equal to the number of connected components of G that are bipartite. 

4, [DH03] The Laplacian and the signless Laplacian of a graph G have the same spectrum if and only 
if G is bipartite. 

5. (Matrix-tree theorem) Let G be a graph with Laplacian matrix Lg, and let cg denote the number 
of spanning trees of G. Then adj(Lg) =cgJ. 

6. Suppose Ng is the incidence matrix of G. Then Ng Né = |Lo| and NENco — 21 = Aji). 

7. Suppose Ny, is an oriented incidence matrix of G. Then NUNi! = Le. 

8. If wy < ... < py are the Laplacian eigenvalues of G, and @, < ... < 7, are the Laplacian 
eigenvalues of G, then 4; = 1, = Oand @; =n — fy42-; fori = 2,... ,n. 

9. [DH03] If uw; < ... < fy are the Laplacian eigenvalues of a graph G with n vertices and m edges, 
then >; ; = 2m < \/nd>; wi (4; — 1) with equality if and only if G is regular. 

10. [DH98] A connected graph G has at most three distinct Laplacian eigenvalues if and only if there 
exist integers jz and 7, such that any two distinct nonadjacent vertices have exactly 4. common 
neighbors, and any two adjacent vertices have exactly 7% common nonneighbors. 

11. If G is k-regular and v ¢ span{]1}, then the following are equivalent: 


= 


* 2 is an eigenvalue of Ag with eigenvector v. 

* k — i is an eigenvalue of Lg with eigenvector v. 

* k +2 is an eigenvalue of |Lg| with eigenvector v. 
* —1— 2d isan eigenvalue of Sg with eigenvector v. 

12. [DH03] Considera simple graph G with n vertices and m edges. Letv, < ... < v, bethe eigenvalues 
of |Lg|, the signless Laplacian of G. Let A, > ... > Am be the eigenvalues of L(G), the line graph 
of G. Then A; = vy-i4) — 2 if 1 <i < min{m, n}, anda; = —2 if min{m,n} <i < m. 

13. [GRO1, p. 298] Let G be a connected graph, let M be a generalized Laplacian of G, and let v be an 
eigenvector for M corresponding to the second smallest eigenvalue of M. Then the subgraph of G 
induced by the vertices corresponding to the positive entries of v is connected. 

14. The Seidel matrices of switching equivalent graphs have the same spectrum. 


FIGURE 28.5 Graphs with cospectral Laplacian matrices. 
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FIGURE 28.6 Graphs with cospectral signless Laplacian matrices. 


Examples: 


1. The Laplacian eigenvalues of the Petersen graph are {0, 2°, 5+}. 

2. The two graphs of Figure 28.5 are nonisomorphic, but the Laplacian matrices have the same 
spectrum. Both Laplacian matrices have 12J as adjugate, so both have 12 spanning trees. They are 
not cospectral with respect to the adjacency matrix because one is bipartite and the other one is not. 

3. Figure 28.6 gives two graphs with cospectral signless Laplacian matrices. They are not cospectral 
with respect to the adjacency matrix because one is bipartite and the other one is not. They also do 
not have cospectral Laplacian matrices because the numbers of components differ. 

4, The eigenvalues of the Laplacian and the signless Laplacian matrix of the path P,, are 2 + 2 cos 2 
(i= 1,...,n). 

5. The complete bipartite graph Ky, 1, is Seidel switching equivalent to the empty graph on n = 1) +n2 
vertices. The Seidel matrices have the same spectrum, being {n — 1,—1"71}. 


28.5 Graph Parameters 


Definitions: 


A subgraph G’ on n’ vertices of a simple graph G is a clique if G’ is isomorphic to the complete graph 
K,. The largest value of n’ for which a clique with n’ vertices exists is called the clique number of G and 
is denoted by w(G). 

An induced subgraph G’ on n’ vertices of a graph G is a coclique or independent set of vertices if G’ 
has no edges. The largest value of n’ for which a coclique with n’ vertices exists is called the vertex indepen- 
dence number of G and is denoted by «(G). Note that the standard notation for the vertex independence 
number of G is a(G), but 1(G) is used here due to conflict with the use of a(G) to denote the algebraic 
connectivity of G in Chapter 36. 

The Shannon capacity @(G) of a simple graph G is defined by ©(G) = sup, V/t(G*) 

A vertex coloring of a graph is a partition of the vertex set into cocliques. A coclique in such a partition 
is called a color class. 

The chromatic number x (G) ofa graph G is the smallest number of color classes of any vertex coloring 
of G. (The chromatic number is not defined if G has loops.) 

For a simple graph G = (V, E), the conductance or isoperimetric number ®(G) is defined to be the 
minimum value of 0(V’)/|V’| over any subset V’ C V with|V’| < |V|/2, where 0(V’) equals the number 
of edges in E with one endpoint in V’ and one endpoint outside V’. 

An infinite family of graphs with constant degree and isoperimetric number bounded from below is 
called a family of expanders. 

A symmetric real matrix M is said to satisfy the Strong Arnold Hypothesis provided there does not 
exist a symmetric nonzero matrix X with zero diagonal, such that MX = 0, Mo X = 0. 

The Colin de Verdiére parameter ;1(G) of a simple graph G is the largest nullity of any generalized 
Laplacian M of G satisfying the following: 


* M has exactly one negative eigenvalue of multiplicity 1. 
* The Strong Arnold Hypothesis. 
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Consider a simple graph G with vertex set {v1,... , v,}. The Lovasz parameter 3 (G) is the minimum 
value of the largest eigenvalue 1,(M) of any real symmetric n x n matrix M = [mj], which satisfies 
mj = lifv; and v; are nonadjacent (including the diagonal). 

Consider a simple graph G with vertex set {v),... ,v,}. The integer 7(G) is defined to be the smallest 
rank of any n x n matrix M (over any field), which satisfies m;; A 0 fori = 1,... ,n and mj = 0, if 
and v; are distinct nonadjacent vertices. 


Facts: 
In the facts below, all graphs are simple. 


1. [Big74, p. 13] A connected graph with r distinct eigenvalues (for the adjacency, the Laplacian or 
the signless Laplacian matrix) has diameter at most r — 1. 

2. [CDS80, pp. 90-91], [God93, p. 83] The chromatic number x(G) of a graph G with adjacency 
eigenvalues A) >... > A, satisfies: 1 — Ay/An < x(G) < 14+A). 

3. [CDS80, p. 88] For a graph G, let m4 and m_ denote the number of nonnegative and nonpositive 
adjacency eigenvalues, respectively. Then 1(G) < min{m,, m_}. 

4. [GRO1, p. 204] If G is a k-regular graph with adjacency eigenvalues A, >... > Ay, then w(G) < 
wee and w(G) < an 

5. [Moh97] Suppose G is a graph with maximum degree A and algebraic connectivity 42. Then the 
isoperimetric number ®(G) satisfies 42/2 < ®(G) < /2(2A — jp). 

6. [HLS99] The Colin de Verdiére parameter j4(G) is minor monotonic, that is, if H is a minor of G, 
then “(H) < “(G). 

7. [HLS99] If G has at least one edge, then (G) = max{j(.H)| H isa component of G}. 

8. [HLS99] The Colin de Verdiére parameter j1(G) satisfies the following: 


° w(G) < 1ifand only if G is the disjoint union of paths. 
° w(G) < 2 if and only if G is outerplanar. 
° w(G) < 3 ifand only if G is planar. 
° u(G) < 4ifand only if G is linklessly embeddable. 
9. (Sandwich theorems) [Lov79], [Hae81] The parameters 7 (G) and n(G) satisfy: ((G) < 0(G) < 

x(G) and (G) < n(G) < x(G). 

10. [Lov79], [Hae81] The parameters ?(G) and n(G) satisfy: }(G- H) = 0(G)0(A) and n(G- H) < 
n(G)n( A). 

11. [Lov79], [Hae81] The Shannon capacity ©(G) ofa graph G satisfies: (G) < O(G), O(G) < B(G), 
and @(G) < n(G). 

12. [Lov79], [Hae81] If G is a k-regular graph with eigenvalues k = A; >... > A,, then v(G) < 
—nh,/(k — 4,). Equality holds if G is strongly regular. 

13. [Lov79] The Lovasz parameter 3 (G) can also be defined as the maximum value of tr(M/J,,), where 
M is any positive semidefinite n x n matrix, satisfying tr(M) = 1 and m;; = 0 if v; and v; are 
adjacent vertices in G. 


Examples: 


1. Suppose G is the Petersen graph. Then 1(G) = 4, 3(G) = 4 (by Facts 9 and 12). Thus, @(G) = 4 
(by Fact 11). Moreover, x(G) = 3, x(G) = 5, u(G) = 5 (take M = Lg — 21), and n(G) = 4 
(take M = Ac + I over the field with two elements). 

2. The isoperimetric number ®(G) of the Petersen graph equals 1. Indeed, ®(G) > 1, by Fact 5, and 
any pentagon gives ®(G) < 1. 

3. w(K,) =n — 1 (take M = —J). 

4. If G is the empty graph with at least two vertices, then 4(G) = 1. (M must be a diagonal matrix 
with exactly one negative entry, and the Strong Arnold Hypothesis forbids two or more diagonal 
entries to be 0.) 
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5. By Fact 12, (Cs) = /5. If (v1,...5¥5) are the vertices of Cs, cyclically ordered, then (v1, 11), 
(V2, V3), (V3, V5)5 (V4, V2), (V5, V4) is. a coclique of size 5 in Cs- Cs. Thus, 1(C5-C5) > 5 and, therefore, 


@(Cs) = V5. 


28.6 Association Schemes 


Definitions: 
A set of graphs Go,... ,Gqg on a common vertex set V = {v1,... ,V,} is an association scheme if the 
adjacency matrices Ao,... , Aq satisfy: 
* Ap =I. 
d 
* Vizo Ai = J- 
* span{Ao,..., Aq} is closed under matrix multiplication. 


The numbers p/ j defined by Aj Aj = a pr ; Ax are called the intersection numbers of the association 
scheme. 

The (associative) algebra spanned by Ao,... , Ag is the Bose—Mesner algebra of the association scheme. 

Consider a connected graph G, = (V, E;) with diameter d. Define G; = (V, E;) to be the graph 
wherein two vertices are adjacent if their distance in G, equals i. If Go,... , Gq is an association scheme, 
then G; is a distance-regular graph. 

Let V’ bea subset of the vertex set V ofan association scheme. The inner distribution a = [ao,... ,aq]" 
of V’ is defined by a;|V’| = c? Ajc, where c is the characteristic vector of V’ (that is, c; = 1ifv; € Vand 
c; = 0 otherwise). 


Facts: 


Facts 1 to 7 below are standard results on association schemes that can be found in any of the following 
references: [BI84], [BCN89], [God93]. 


1. Suppose Go,... , Gg is an association scheme. For any three integers 1, j,k € {0,...,d} and for 
any two vertices x and y adjacent in G;, the number of vertices z adjacent to x in G; and to y in 
Gj; equals the intersection number pi. In particular, G; is regular of degree k; = p?. (i #0). 

2. The matrices of a Bose—Mesner algebra A can be diagonalized simultaneously. In other words, there 
exists a nonsingular matrix S such that $ AS! is a diagonal matrix for every A € A. 

3. A Bose—Mesner algebra has a basis {Ey = +J, E,,..., Eq} of idempotents, that is, E; E; = 6;,; Ej 
(4;,; 1s the Kronecker symbol). 

4, The change-of-coordinates matrix P = [pj;;] defined by Aj = }°; pij £; satisfies: 


* pjij is an eigenvalue of A; with eigenspace range(E;). 
* pio = 1, poi = k; (the degree of G; (i # 0)). 
* nkj(P~!) j; = mj pij, where m; = rank(E;) (the multiplicity of eigenvalue p;;). 
5. (Krein condition) The Bose—Mesner algebra of an association scheme is closed under Hadamard 


multiplication. The numbers qs j defined by Ej o Ej = >), qi j Ek are nonnegative. 
6. (Absolute bound) The multiplicities mo = 1,m),...,mq of an association scheme satisfy 


a m <mym; and S my < m;(m; + 1)/2. 
k:gi|>0 k:qk,>0 


7. A connected strongly regular graph is distance-regular with diameter two. 
8. [BCN839, p. 55] Let V’ be a subset of the vertex set V of an association scheme with change-of- 
coordinates matrix P. The inner distribution a of V’ satisfies a? P~! > 0. 
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Examples: 


1. The change-of-coordinates matrix P of a strongly regular graph with eigenvalues k, v2, and v3 is 
equal to 


1 k n-k-1 
1 V2 —vy,-1 


1 V3 —v3 — 1 


2. Astrongly regular graph with parameters (28, 9, 0, 4) cannot exist, because it violates Facts 5 and 6. 

3. The Hamming association scheme H(d,q) has vertex set V = Q4, the set of all vectors with d 
entries from a finite set Q of size q. Two such vectors are adjacent in G; if they differ in exactly 
i coordinate places. The graph Gj is distance-regular. The matrix P of a Hamming association 
scheme can be expressed in terms of Kravéuk polynomials, which gives 


j WGN YT hes 
pi = ¥-(-DF(q - vale ) 
: » k ja—k 


4. An error correcting code with minimum distance 6 is a subset V’ of the vertex set V of a Hamming 
association scheme, such that V’ induces a coclique in Gj,... , Gs_y. Ifa is the inner distribution 
of V’, then do 1, a, see as—1 0, and |V’| = )°; a;. Therefore, by Fact 8, the linear 
programming problem “Maximize )~;..; a;, subject to a’ P~! > 0 (with aj) = 1 anda, =--- = 
as—, = 0)” leads to an upper bound for the size of an error correcting code with given minimum 
distance. This bound is known as Delsarte’s Linear Programming Bound. 

5. The Johnson association scheme J (d, 2) has as vertex set V all subsets of size d of a set of size € 
(£ > 2d). Two vertices are adjacent in G; if the intersection of the corresponding subsets has size 
d — i. The graph G, is distance-regular. The matrix P of a Johnson association scheme can be 
expressed in terms of Eberlein polynomials, which gives: 


j Naa (tad 4 
j= Sea . ; 
w= EGGS) 
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Digraphs and Matrices 


Directed graphs, often called digraphs, have much in common with graphs, which were the subject of the 
previous chapter. While digraphs are of interest in their own right, and have been the subject of much 
research, this chapter focuses on those aspects of digraphs that are most useful to matrix theory. In par- 
ticular, it will be seen that digraphs can be used to understand how the zero—nonzero structure of square 


matrices affects matrix products, determinants, inverses, and eigenstructure. Basic material on digraphs 


and their adjacency matrices can be found in many texts on graph theory, nonnegative matrix theory, 
or combinatorial matrix theory. For all aspects of digraphs, except their spectra, see [BG00]. Perhaps the 
most comprehensive single source for results, proofs, and references to original papers on the interplay 
between digraphs and matrices is [BR91, Chapters 3 and 9]. Readers preferring a matrix analytic rather 
than combinatorial approach to irreducibility, primitivity, and their consequences, should consult [BP94, 


Chapter 2]. 


29.1 Digraphs 


Definitions: 


A directed graph [T = I'(V, E) consists of a finite, nonempty set V of vertices (sometimes called nodes), 
together with a multiset E of elements of V x V, whose elements are called arcs (sometimes called edges, 


directed edges, or directed arcs). 
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FIGURE 29.1 


A loop is an arc of the form (v,v) for some vertex v. 

If there is more than one arc (u,v) for some u and v in V, then I is called a directed multigraph. 

If there is at most one arc (u, v) for each u and v in V, then T is called a digraph. 

If the digraph I contains no loops, then I is called a simple digraph. 

A weighted digraph is a digraph I with a weight function w : E — F, where the set F is often the real 
or complex numbers. 

A subdigraph I’ of a digraph I is a digraph T’ = T’’(V’, E’) such that V’ C Vand E’ C E. 

A proper subdigraph I’ of a digraph I is a subdigraph of T such that V’ C Vor E’ C E. 

If V’ is a nonempty subset of V, then the induced subdigraph of I induced by V’ is the digraph with 
vertex set V’ whose arcs are those arcs in E that lie in V’ x V’. 

A walk is a sequence of arcs (Vo,V1), (V1, V2),-.- »(Vk—1, Vk), Where one or more vertices may be 
repeated. 

The length of a walk is the number of arcs in the walk. (Note that some authors define the length to be 
the number of vertices rather than the number of arcs.) 

A simple walk is a walk in which all vertices, except possibly vo and vx, are distinct. (Note that some 
authors use path to mean what we call a simple walk.) 

A cycle is a simple walk for which vp = vz. A cycle of length k is called a k-cycle. 

A generalized cycle is either a cycle passing through all vertices in V or else a union of cycles such that 
every vertex in V lies on exactly one cycle. 

Lett = ['(V, E) bea digraph. The undirected graph G associated with the digraph I’ is the undirected 
graph with vertex set V, whose edge set is determined as follows: There is an edge between vertices u and 
v in G ifand only if at least one of the arcs (u,v) and (v, u) is present in I’. 

The digraph I is connected if the associated undirected graph G is connected. 

The digraph I is a tree if the associated undirected graph G is a tree. 

The digraph I’ is a doubly directed tree if the associated undirected graph is a tree and if whenever 
(i, j) isan arc inT, (j,7) is also an arc inT. 


Examples: 


1. The key distinction between a graph on a vertex set V and a digraph on the same set is that for a 
graph we refer to the edge between vertices u and v, whereas for a digraph, we have two arcs, the 
arc from u to v and the arc from v to u. Thus, there is one connected, simple graph on two vertices, 
K» (see Figure 29.1a), but there are three possible connected, simple digraphs on two vertices 
(see Figure 29.1b). Note that all graphs in Figure 29.1 are trees, and that the graph in Figure 29.1a 
is the undirected graph associated with each of the digraphs in Figure 29.1b. The third graph in 
Figure 29.1b is a doubly directed tree. 
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2. Let I’ be the digraph in Figure 29.2. Then (1,1), 
(1,1), (1,3), (3,1), (1,2) is a walk of length 5 
from vertex 1 to vertex 2. (1, 2), (2,3) isa simple 
walk of length 2. (1, 1) is a 1-cycle; (1, 3), (3, 1) 
is a 2-cycle; and (1,2), (2,3), (3,1) is a 3-cycle. 
(1, 1), (2, 3), (3, 2) and (1, 2), (2, 3), (3, 1) are two 
generalized cycles. Not all digraphs contain gen- 
eralized cycles; consider the digraph obtained by 3 2 
deleting the arc (2,3) from I, for example. Un- 
less we are emphasizing a particular vertexona FIGURE29.2 
cycle, such as all cycles starting (and ending) at 
vertex v, we view cyclic permutations of a cycle 
as equivalent. Thatis, in Figure 29.2, we would speak of the 3-cycle, although technically (1, 2), (2, 3), (3, 1); 
(2, 3), (3, 1), (1, 2); and (3, 1), (1, 2), (2, 3) are distinct cycles. 


29.2 The Adjacency Matrix of a Directed Graph 
and the Digraph of a Matrix 


If is a directed graph on n vertices, then there is a natural way that we can record the arc information 
for T in ann x n matrix. Conversely, if A isan n x n matrix, we can naturally associate a digraph T on n 
vertices with A. 


Definitions: 


Let I be a digraph with vertex set V. Label the vertices in V as vj, v2,... , Vn. Once the vertices have been 
ordered, the adjacency matrix for I’, denoted Ar, is the 0, 1-matrix whose entries a;; satisfy: aj; = 1 if 
(v;,v;) is an arc in I’, and aj; = 0 otherwise. When the set of vertex labels is {1,2,... ,n}, the default 
labeling of the vertices is vj = i forl1 <i <n. 

Let A be an n x n matrix. Let V be the set {1,2,... ,m}. Construct a digraph denoted (A) on V as 
follows. For each i and j in V, let (i, j) be an arc in P' exactly when a;; 4 0. P(A) is called the digraph of 
the matrix A. Commonly I’ is viewed as a weighted digraph with weight function w((i, j)) = aj; for all 
(i, j) with ij # 0. 


Facts: [BR91, Chap. 3] 


1. Ifa digraph H is obtained from a digraph I by adding or removing an arc, then Ay is obtained by 
changing the corresponding entry of Ap to a 1 or a 0, respectively. If a digraph H is obtained from 
a digraph I by deleting the i” vertex in the ordered set V and by deleting all arcs in E containing 
the i” vertex, then Ay = Ar(i). That is, Ay is obtained by deleting rowi and column i of Ar. 

2. Given one ordering of the vertices in V, any other ordering of those vertices is simply a permutation 
of the original ordering. Since the rows and columns of A = Ar are labeled by the ordered vertices, 
reordering the vertices in I corresponds to simultaneously permuting the rows and columns of 
A. That is, if P is the permutation matrix corresponding to a permutation of the vertices of V, 
then the new adjacency matrix is PAP’. Since P’ = P~! for a permutation matrix, all algebraic 
properties preserved by similarity transformations are invariant under changes of the ordering of 
the vertices. 

3. Let P bea digraph with vertex set V. The Jordan canonical form of an adjacency matrix for I is inde- 
pendent of the ordering applied to the vertices in V. Consequently, all adjacency matrices for I’ have 
the same rank, trace, determinant, minimum polynomial, characteristic polynomial, and spectrum. 

4. If Aisann x n matrix and if vj =i for 1 <i <n, then A and Aya) have the same zero—nonzero 
pattern and, hence, ['(A) = T'(Arya)). 
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Examples: 1 
1. For the digraph I given in Figure 29.3, if 2 

we order the vertices as vj = i fori = 

1,2,...,7, then 
0 1 1 1°00 0 4 
, 0 00 1 0 0 
0 00 0 0 1 0 

A=Ar=j0 01 00 0 0 

| 0 100 0 0 0 
° Oo1001 1 FIGURE 29.3 
0 00 00 0 1 


If we reorder the vertices in the digraph I’ given in Figure 29.3 so that v1, v2, -- 


1, 2,5, 4, 3, 6, 7, then the new adjacency matrix is 


@. 2.) ae) ao: [hag 
hee ce abe aie: 
Sa eee ree 
B=Ar=]0 00]0/1 0! 0 
6-0: 0 |) 6} O14) 6 
Gro ip ge) Cae ae || 
0.0.) -0 |-.O O72 


2. If Ais the 3 x 3 matrix 


eo 

II 
© ow 

° 

| 

je 

= 


then I'(A) is the digraph given in Figure 29.2. Up to permutation similarity, 


29.3. Walk Products and Cycle Products 


- , v7 is the sequence 


For a square matrix A, a124@3 is nonzero exactly when both aj2 and a3 are nonzero. That is, exactly when 
both (1,2) and (2,3) are arcs in P(A). Note also that aja); is one summand in (A’),3. Consequently, 
there is a close connection between powers of a matrix A and walks in its digraph I’(A). In fact, the signs 
(complex arguments) of walk products play a fundamental role in the study of the matrix sign patterns 
for real matrices (matrix ray patterns for complex matrices). See Chapter 33 [LHE94] or [Stu03]. 


Definitions: 


Let A be an n x n matrix. Let W given by (v9, v1), (V1, V2),.-- 5 (Vk—-1 Ve) be a walk in (A). The walk 


product for the walk W is 


k 
[4-093 
j=l 
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and is often denoted by [| aij. This product is a generic summand of the (vo, vz)-entry of A“. If 
$1,$2... 5S, are scalars, [[ ys; denotes the ordinary product of the s; over the index set vo, v1, V2... 5 
Vk—1) Vk. 

If W is a cycle in the directed graph I'(A), then the walk product for W is called a cycle product. 

Let A be an n x n real or complex matrix. Define | A| to be the matrix obtained from A by replacing aj; 
with |a;;| for alli and j. 


Facts: 


1. Let A be a square matrix. The walk W given by (v9, v1), (V1, V2)5.-- 5 (Ve—15 Ve) Occurs in T'(A) 
exactly when 4, y,4y,1, °° * @yj_,», 18 nonzero. 

2. [BR91, Sec. 3.4] [LHE94] Let A be a square matrix. For each positive integer k, and for all i and 
j» the (i, j)-entry of A* is the sum of the walk products for all length k walks in P(A) from i to 
j- Further, there is a walk of length k from i to j in P(A) exactly when the (i, j)-entry of | A|* is 
nonzero. 

3. [BR91, Sec. 3.4] [LHE94] Let A be a square, real or complex matrix. For all positive integers k, 

T(A*) isa subdigraph of P({Al*). Further, 7(A*) is a proper subgraph of ral) exactly when 

additive cancellation occurs in summing products for length k walks from some i to some j. 

4, [LHE94] Let A be a square, real matrix. The sign pattern of the k"" power of A is determined solely 

by the sign pattern of A when the signs of the entries in A are assigned so that for each ordered pair 

of vertices, all products of length k walks from the first vertex to the second have the same sign. 

5. [FP69] Let A and B be irreducible, real matrices with [(A) = I'(B). There exists a nonsingular, 
real diagonal matrix D such that B = DAD7 if and only if the cycle product for every cycle in 
T'(A) equals the cycle product for the corresponding cycle in T'(B). 

6. Let A be an irreducible, real matrix. There exists a nonsingular, real diagonal matrix D such that 
DAD7' is nonnegative if and only if the cycle product for every cycle in (A) is positive. 


Examples: 


1 1 
EBs 1 1? then (A*) , = a11412 + 412422 = (1)(1) + (1)(—1) = 0, whereas (IAI?) =e: 


2. If A is the matrix in Example 2 of the previous section, then (A) contains four cycles: the loop 
(1,1); the two 2-cycles (1,3), (3,1) and (2,3), (3,2); and the 3-cycle (1,2), (2,3), (3,1). Each of 
these cycles has a positive cycle product and using D = diag(1,—1,1), DAD™ is nonnegative. 


29.4 Generalized Cycle Products 


If the matrix A is 2 x 2, then det(A) = aja. — a)2a2). Assuming that the entries of A are nonzero, 
the two summands aj;4@2) and aj7a2 are exactly the walk products for the two generalized cycles of 
I'(A). From Chapter 4.1, the determinant of an n x n matrix A is the sum of all terms of the form 
(—1)8") ay 5, aj, +++ Anj,, Where o = (ji, jz...» jn) is a permutation of the ordered set {1,2,... ,n}. 
Such a summand is nonzero precisely when (1, j1), (2; J2),-++ 5 (1 jn) are all arcs in (A). For this set 
of arcs, there is exactly one arc originating at each of the n vertices and exactly one arc terminating at 
each of the n vertices. Hence, the arcs correspond to a generalized cycle in (A). See [BR91, Sect. 9.1]. 
Since the eigenvalues of A are the roots of det(AI — A), it follows that results connecting the cycle 
structure of a matrix to its determinant should play a key role in determining the spectrum of the 
matrix. For further results connecting determinants and generalized cycles, see [MOD89] or [BJ86]. 
Generalized cycles play a crucial role in the study of the nonsingularity of sign patterns. (See Chap- 
ter 33 or [Stu91]). In general, there are fewer results for the spectra of digraphs than for the spectra 
of graphs. There are generalizations of Gersgorin’s Theorem for spectral inclusion regions for complex 
matrices that depend on directed cycles. (See Chapter 14.2 [Bru82] or [BR91, Sect. 3.6], or especially, 
[Var04]). 
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Definitions: 
Let A be ann x n matrix. Leto = ; ; es i be a permutation of the ordered set {1,2,... , }. 
i dee * Te 
When (i, j;) is an arc in (A) for each 1, the cycle or vertex disjoint union of cycles with arcs (1, j;), 
(2, j2),..- 5 (mM; jn) is called the generalized cycle induced by o. 
The product of the cycle products for the cycle(s) comprising the generalized cycle induced by a is 


called the generalized cycle product corresponding to o. 


Facts: 


1. The entries in A that correspond to a generalized cycle are a diagonal of A and vice versa. 

2. If o is a permutation of the ordered set {1,2,...,m}, then the nonzero entries of the n x n 
permutation matrix P corresponding to o are precisely the diagonal of P corresponding to the 
generalized cycle induced by o. 

3. [BR91, Sec. 9.1] Let A be an n x n real or complex matrix. Then det(A) is the sum over all 
permutations of the ordered set {1,2,... ,n} of all of the signed generalized cycle products for 
I'(A) where the sign of a generalized cycle is determined as (—1)"-*, where k is the number of 
disjoint cycles in the generalized cycle. If [(A) contains no generalized cycle, then A is singular. 
If (A) contains at least one generalized cycle, then A is nonsingular unless additive cancellation 
occurs in the sum of the signed generalized cycle products. 

4. [Cve75] [Har62] Let A be ann x n real or complex matrix. The coefficient of x" in det(xI — A) 
is the sum over all induced subdigraphs H of (A) on k vertices of the signed generalized cycle 
products for H. 

5. [BP94, Chap. 3], [BR97, Sec. 1.8] Let A be an n x n real or complex matrix. Let g be the greatest 
common divisor of the lengths of all cycles in T(A). Then det(xJ — A) = x* p(x) for some 
nonnegative integer k and some polynomial p(z) with p(0) 4 0. Consequently, the spectrum of 
A is invariant under = rotations of the complex plane. Further, if A is nonsingular, then g divides 
n, and det(xI — A) = p(x®) for some polynomial p(z) with p(0) = (—1)” det(A). 


Examples: 


1. If A is the matrix in Example 2 of the previous section, then (A) contains two generalized 
cycles — the loop (1, 1) together with the 2-cycle (2, 3), (3,2); and the 3-cycle (1, 2), (2, 3), (3, 1). 
The corresponding generalized cycle products are (3)(—11)(—6) = 198 and (—2)(—11)(9) = 198, 
with corresponding signs —1 and 1, respectively. Thus, det(A) = 0 is a consequence of additive 
cancellation. 


1 0 
2. If A= Root? then the only cycles in I'(A) are loops, so g = 1. The spectrum of A is clearly 


invariant under 2 rotations, but it is also invariant under rotations through the smaller angle of z. 


29.5 Strongly Connected Digraphs and Irreducible Matrices 


Irreducibility of a matrix, which can be defined in terms of permutation similarity (see Section 27.3), 
and which Frobenius defined as an algebraic property in his extension of Perron’s work on the spectra 
of positive matrices, is equivalent to the digraph property of being strongly connected, defined in this 
section. Today, most discussions of the celebrated Perron—Frobenius Theorem (see Chapter 9) use digraph 
theoretic terminology. 


Definitions: 


Vertex u has access to vertex v in a digraph I’ if there exists a walk in I’ from u to v. By convention, every 
vertex has access to itself even if there is no walk from that vertex to itself. 

If u and v are vertices in a digraph I’ such that u has access to v, and such that v has access to u, then u 
and v are access equivalent (or u and v communicate). 
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Access equivalence is an equivalence relation on the vertex set V of I that partitions V into access 
equivalence classes. 

If V; and V2 are nonempty, disjoint subsets of V, then V, has access to V> if some vertex in v; in Vj has 
access in I’ to some vertex v2 in V>. 

For a digraph I’, the subdigraphs induced by each of the access equivalence classes of V are the strongly 
connected components of I. 

When all of the vertices of T’ lie in a single access equivalence class, T' is strongly connected. 

Let Vi, V2,..., Vk be the access equivalence classes for some digraph I. Define a new digraph, R(T), 
called the reduced digraph (also called the condensation digraph) for I as follows. Let W = {1,2,... ,k} 
be the vertex set for R(T’). Ifi, 7 € Wwithi # j, then (i, j) is an arc in R(T) precisely when V; has access 
to Vj. 


Facts: [BR91, Chap. 3] 


1. [BR91, Sec. 3.1] A digraph I is strongly connected if and only if there is a walk from each vertex in 
I’ to every other vertex in T’. 

2. The square matrix A is irreducible if and only if (A) is strongly connected. A reducible matrix A 
is completely reducible if (A) is a disjoint union of two or more strongly connected digraphs. 

3. [BR91, Sec. 3.1] Suppose that V; and V) are distinct access equivalence classes for some digraph 
I’. If any vertex in V, has access to any vertex in V2, then every vertex in V, has access to every 
vertex in V>. Further, exactly one of the following holds: V, has access to V2, V2 has access to 
V\, or neither has access to the other. Consequently, access induces a partial order on the access 
equivalence classes of vertices. 

4, [BR91, Lemma 3.2.3] The access equivalence classes for a digraph I can be labelled as Vi, V2,... , Vk 

so that whenever there is an arc from a vertex in V; to a vertex in V;,i < j. 

5. [Sch86] IfT is a digraph, then R(T) is a simple digraph that contains no cycles. Further, the vertices 

in R(T) can always be labelled so that if (i, 7) is an arc in R(T), theni < j. 

6. [BR91, Theorem 3.2.4] Suppose that I is not strongly connected. Then there exists at least one 

ordering of the vertices in V so that Ar is block upper triangular, where the diagonal blocks of Ar 

are the adjacency matrices of the strongly connected components of I. 

7. [BR91, Theorem 3.2.4] Let A be a square matrix. Then A has a Frobenius normal form. (See 
Chapter 27.3.) 

8. The Frobenius normal form of a square matrix A is not necessarily unique. The set of Frobenius 
normal forms for A is preserved by permutation similarities that correspond to permutations 
that reorder the vertices within the access equivalence classes of (A). If B is a Frobenius normal 
form for A, then all the arcs in R(I'(B)) satisfy (i,j) is an edge implies i < j. Let o be any 
permutation of the vertices of R(I’(B)) such that (o (i),0(j)) is an edge in R(I'(B)) implies 
o(i) < o(j). Let B be block partitioned by the access equivalence classes of ['(B). Applying the 
permutation similarity corresponding to o to the blocks of B produces a Frobenius normal form 
for A. All Frobenius normal forms for A are produced using combinations of the above two types 
of permutations. 

9. [BR91, Sect. 3.1] [BP94, Chap. 3] Let A be ann x n matrix for n > 2. The following are equivalent: 


(a) A is irreducible. 

(b) T'(A) is strongly connected. 

(c) For each i and j, there is a positive integer k such that (Al); > 0. 
(d) There does not exist a permutation matrix P such that 


Au | Av 


PAP! = 
0 | Ax 


> 


where Aj; and Az) are square matrices. 
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10. 


11. 


12. 


Let A be a square matrix. A is completely reducible if and only if there exists a permutation matrix 
P such that PAP? is a direct sum of at least two irreducible matrices. 

All combinations of the following transformations preserve irreducibility, reducibility, and 
complete reducibility: Scalar multiplication by a nonzero scalar; transposition; permutation simi- 
larity; left or right multiplication by a nonsingular, diagonal matrix. 

Complex conjugation preserves irreducibility, reducibility, and complete reducibility for square, 


complex matrices. 


Examples: 


1. 


29.6 Primitive Digraphs and Primitive Matrices 


In the digraph I in Figure 29.3, vertex 4 has 
access to itself and to vertices 3, 6, and 7, 
but not to any of vertices 1, 2, or 5. For 
I’, the access equivalence classes are: Vj = 
{1,2,5}, Vo = {3,6}, V3 = {4}, and Vy = 
{7}. The strongly connected components of 
l are given in Figure 29.4, and R(T) is given 
in Figure 29.5. Ifthe access equivalence classes 
for I’ are relabeled so that V) = {4} and 
V3; = {3, 6}, then the labels on vertices 2 and 
3 switch in the reduced digraph given in Fig- 
ure 29.5. With this labeling, if (i, 7) is an arc 
in the reduced digraph, theni < j. 


If A and B are the adjacency matrices of 
Example 1 of Section 29.2, then A(3, 4, 6,7) = 
A[1, 2, 5] isthe adjacency matrix for the largest 
strongly connected component of the digraph 
I in Figure 29.3 using the first ordering of 
V; and using the second ordering of V, B is 
block-triangular and the irreducible, diago- 
nal blocks of B are the adjacency matrices for 
each of the four strongly connected compo- 
nents of I’. B is a Frobenius normal form for 
A. 


form. 


FIGURE 29.4 


1 


FIGURE 29.5 
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3. The matrix A in Example 2 of Section 29.2 is irreducible, hence, it is its own Frobenius normal 


Primitive matrices, defined in this section, are necessarily irreducible. Unlike irreducibility, primitivity 
depends not just on the matrix A, but also its powers, and, hence, on the signs of the entries of the original 
matrix. Consequently, most authors restrict discussions of primitivity to nonnegative matrices. Much work 
has been done on bounding the exponent of a primitive matrix; see [BR91, Sec. 3.5] or [BP94, Sec. 2.4]. 
One consequence of the fifth fact stated below is that powers of sparse matrices and inverses of sparse 


matrices can experience substantial fill-in. 


Definitions: 


A digraph I" with at least two vertices is primitive if there is a positive integer k such that for every pair of 
vertices u and v (not necessarily distinct), there exists at least one walk of length k from u to v. A digraph 


on a single vertex is primitive if there is a loop on that vertex. 
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The exponent of a digraph I’ (sometimes called the index of primitivity) is the smallest value of k that 
works in the definition of primitivity. 

A digraph I is imprimitive if it is not primitive. This includes the simple digraph on one vertex. 

If A is a square, nonnegative matrix such that (A) is primitive with exponent k, then A is called a 
primitive matrix with exponent k. 


Facts: 


1. A primitive digraph must be strongly connected, but not conversely. 

2. A strongly connected digraph with at least one loop is primitive. 

3. [BP94, Chap. 3] [BR91, Sections 3.2 and 3.4] Let I’ be a strongly connected digraph with at least 
two vertices. The following are equivalent: 


(a) IT is primitive. 
(b) The greatest common divisor of the cycle lengths for PF is 1 (i.e., P is aperiodic, cf. Chapter 9.2). 


(c) There is a smallest positive integer k such that for each t > k and each pair of vertices u and v 
in T, there is a walk of length t from u to v. 


4, [BR91, Sect. 3.5] Let P be a primitive digraph with n > 2 vertices and exponent k. Then 
(a) k<(n—1) +1. 
(b) Ifs is the length of the shortest cycle in’, then k < n+ s(n — 2). 
(c) Iff has p > 1 loops, then k < 2n— p—1. 


5. [BR91, Theorem 3.4.4] Let A be an n x n nonnegative matrix with n > 2. The matrix A is primitive 
if and only if there exists a positive integer k such that A* is positive. When such a positive integer 
k exists, the smallest such k is the exponent of A. Further, if At is positive, then A? is positive for 
all integers h > k. A nonnegative matrix A with the property that some power of A is positive is 
also called regular. See Chapter 9 and Chapter 54 for more information about primitive matrices 
and their uses. 

6. [BP94, Chap. 3] If A is an irreducible, nonnegative matrix with positive trace, then A is primitive. 

7. [BP94, Chap. 6] Let A bea nonnegative, tridiagonal matrix with all entries on the first superdiagonal 
and on the first subdiagonal positive, and at least one entry on the main diagonal positive. Then A 
is primitive and, hence, some power of A is positive. Further, ifs > p(A) where p(A) is the spectral 
radius of A, then the tridiagonal matrix s J — A is a nonsingular M-matrix with a positive inverse. 


Examples: 
1. The digraph I in Figure 29.2 is primitive with exponent 3; the strongly connected digraph in Figure 
29.1b is not primitive. 
1 1 1 -l ‘ 
2. Let Ay = iy and let A, = I 34 . Note that A; = |A)|. Clearly, P(A) = '(A2) is an 
irreducible, primitive digraph with exponent 1. For all positive integers k, A is positive, so it makes 
sense to call A; a primitive matrix. In contrast, Ae = 0 for all integers k > 2. 


29.7 Irreducible, Imprimitive Matrices 
and Cyclic Normal Form 


While most authors restrict discussions of matrices with primitive digraphs to nonnegative matrices, many 
authors have exploited results for imprimitive digraphs to understand the structure of real and complex 
matrices with imprimitive digraphs. 
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Definitions: 


Let A be an irreducible n x n matrix with n > 2 such that I'(A) is imprimitive. The greatest 
common divisor g > 1 of the lengths of all cycles in (A) is called the index of imprimitivity of A 
(or period of A). 

If there is a permutation matrix P such that 


0 A, O 0 
0 oO A 0 

PAP’ = ; 
0 oOo oO Ag-1 
Ag 0 0 0 


where each of the diagonal blocks is a square zero matrix, then the matrix PAP’ is called a cyclic normal 
form for A. 
By convention, when A is primitive, A is said to be its own cyclic normal form. 


Facts: 


1. [BP94, Sec. 2.2] [BR91, Sections 3.4] [Min88, Sec. 3.3-3.4] Let A be an irreducible matrix with 
index of imprimitivity g > 1. Then there exists a permutation matrix P such that PAP? isa cyclic 
normal form for A. Further, the cyclic normal form is unique up to cyclic permutation of the blocks 
Aj and permutations within the partition sets of V of (A) induced by the partitioning of PAP’. 
Finally, if A is real or complex, then | A;||A2|--- |Ag| is irreducible and nonzero. 

2. [BP94, Sec. 3.3] [BR91, Sec. 3.4] If A is an irreducible matrix with index of imprimitivity g > 1, 
and if there exists a permutation matrix P and a positive integer k such that 


0 A, O 0 
0 0 A 0 

PAP! = : 
0 0 O - Agy 
Ar. 0 0 0 


where each diagonal block is square zero matrix, then k divides g. Conversely, if A is real or complex, 
if PAP? has the specified form for some positive integer k, if PAP? has no zero rows and no zero 
columns, and if | A;| | A2|---|Ax| is irreducible, then A is irreducible, and k divides g. 

3. [BR91, Sec. 3.4] Let A be an irreducible, nonnegative matrix with index of imprimitivity g > 1. 
Let m be a positive integer. Then A” is irreducible if and only if m and g are relatively prime. If 
A" is reducible, then it is completely reducible, and it is permutation similar to a direct sum of r 
irreducible matrices for some positive integer r. Further, either each of these summands is primitive 
(when g/r = 1), or each of these summands has index of imprimitivity g/r > 1. 

4, [Min88, Sec. 3.4] Let A be an irreducible, nonnegative matrix in cyclic normal form with index of 
imprimitivity g > 1. Suppose that for 1 < i < k —1, A; is nj x nj4, and that A, is ng x ny. Let 
k=min (n 1p M25-+6 5 Ng) . Then 0 is an eigenvalue for A with multiplicity at least n — gk; and if A 
is nonsingular, then each nj = n/g. 

5. [Min88, Sec. 3.4] If A is an irreducible, nonnegative matrix in cyclic normal form with index 
of imprimitivity g, then for j = 1,2,...,g, reading the indices modulo g, Bj = Taf Ai 
is irreducible. Further, all of the matrices B; have the same nonzero eigenvalues. If the nonzero 
eigenvalues (not necessarily distinct) of By are @),@2,... ; @m for some positive integer m, then the 
spectrum of A consists of 0 with multiplicity n — gm together with the complete set of g™ roots of 
each of the @;. 
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6. Let A bea square matrix. Then A has a Frobenius normal form for which each irreducible, diagonal 
block is in cyclic normal form. 

7. Let A bea square matrix. If A is reducible, then the spectrum of A (which is a multiset) is the union 
of the spectra of the irreducible, diagonal blocks of any Frobenius normal form for A. 

8. Explicit, efficient algorithms for computing the index of imprimitivity and the cyclic normal form 
for an imprimitive matrix and for computing the Frobenius normal form for a matrix can be found 
in [BR91, Sec. 3.7]. 

9. All results stated here for block upper triangular forms have analogs for block lower triangular 


forms. 
Examples: 
0 A, 0 1 
1. If Aisthe 4 x 4 matrix A = Me ; , where A; = In and A; = i , then A and A; A, = A> 
2 


are irreducible but g ¥ 2. In fact, g = 4 since A is actually a permutation matrix corresponding to 
the permutation (1324). Also note that when Ay = A) = 1, A is completely reducible since (A) 
consists of two disjoint cycles. 


0 -1l1 0 -1l 
: : ‘ : 6 0 3 . : 
2. If M is the irreducible matrix M = ; Blig , then g = 2, and using the permutation 
6 3 0 
0 0 0 1 
0) 10-0 4 
matrix Q = Berio ae cal? N = QMQ’ = is a cyclic normal form for M. 
1 0 0 0 


3 6] |2 2 2: FD), : 
Note that |.N;| |N2| = ; ] é i — E "| is irreducible even though N; N) = 


0 O;. 
od 
not irreducible. 
3. The matrix B in Example 1 of Section 29.2 is a Frobenius normal form for the matrix A in that 
example. Observe that B,, is imprimitive with g = 2, but Bj; is not in cyclic normal form. The 
remaining diagonal blocks, Bj. and B33, have primitive digraphs, hence, they are in cyclic normal 


0 1 0 0/3 6 
form.LetQ=]1 0 0]|.Then QB,,Q? = 2 | 0 0] isacyclic normal form for B,,. Using 
0 0 1 -1;0 0O 


the permutation matrix P = Q@ 1; ®L@ |, PBP! is a Frobenius normal form for A with 
each irreducible, diagonal block in cyclic normal form. 


2 1 +0 2 od 
4. LetA=1|]1 2 0] andletB=J]1 2 0] . Observethat A and B are each in Frobenius normal 
0 0 3 0 0 3 


form, each with two irreducible, diagonal blocks, and that Aj; = B,; and Ay, = By). Consequently, 
o(A) = 0(B) = o(Aj,) Uo(Az) = {1,3, 3}. However, B has only one independent eigenvector 
for eigenvalue 3, whereas A has two independent eigenvectors for eigenvalue 3. The underlying 
cause for this difference is the difference in the access relations as captured in the reduced digraphs 
R(T(A)) (two isolated vertices) and R(I'(B)) (two vertices joined by a single arc). The role that 
the reduced digraph of a matrix plays in connections between the eigenspaces for each of the 
irreducible, diagonal blocks of a matrix and those of the entire matrix is discussed in [Sch86] and 
in [BP94, Theorem 2.3.20]. For connections between R(I'(A)) and the structure of the generalized 
eigenspaces for A, see also [Rot75]. 
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29.8 Minimally Connected Digraphs and Nearly 
Reducible Matrices 


Replacing zero entries in an irreducible matrix with nonzero entries preserves irreducibility, and equiva- 
lently, adding arcs to a strongly connected digraph preserves strong connectedness. Consequently, it is of 
interest to understand how few nonzero entries are needed in a matrix and in what locations to guarantee 
irreducibility; or equivalently, how few arcs are needed in a digraph and between which vertices to guar- 
antee strong connectedness. Except as noted, all of the results in this section can be found in both [BR91, 
Sec. 3.3] and [Min88, Sec. 4.5]. Further results on nearly irreducible matrices and on their connections to 
nearly decomposable matrices can be found in [BH79]. 


Definitions: 


A digraph is minimally connected if it is strongly connected and if the deletion of any arc in the digraph 
produces a subdigraph that is not strongly connected. 


Facts: 


1. A digraph I is minimally connected if and only if Ar is nearly reducible. 

. A matrix A is nearly reducible if and only if P(A) is minimally connected. 

. The only minimally connected digraph on one vertex is the simple digraph on one vertex. 

. The only nearly reducible 1 x 1 matrix is the zero matrix. 

[BR91, Theorem 3.3.5] Let T be a minimally connected digraph on n vertices with n > 2. Then T 

has no loops, at least n arcs, and at most 2(m — 1) arcs. When I has exactly n arcs, I is an n-cycle. 

When IT has exactly 2(n — 1) arcs, I is a doubly directed tree. 

6. Let A be an n x n nearly reducible matrix with n > 2. Then A has no nonzeros on its main 
diagonal, A has at least n nonzero entries, and A has at most 2(n — 1) nonzero entries. When A 
has exactly n nonzero entries, (A) is an n-cycle. When A has exactly 2(n — 1) nonzero entries, 
T(A) is a doubly directed tree. 

7. [Min88, Theorem 4.5.1] Let A be ann x n nearly reducible matrix with n > 2. Then there exists a 
positive integer m and a permutation matrix P such that 


Wok wd 


° 1 O --- 0| OF 

00 1 0 | oF 

feat cea ie vag. ies Pigs EYE 
0 0 0 41/07 
0 0 0 O|v 
u 0 0 O0O| 8B 


where the upper left matrix is m x m, B is nearly reducible, 0 is a (n — m) x 1 vector, both of the 
vectors u and v are (n — m) x 1, and each of u and v contains a single nonzero entry. 


Examples: 


1. Let I be the third digraph in Figure 29.1b. Then T° is minimally connected. The subdigraph I’ 
obtained by deleting arc (1, 2) from T is no longer strongly connected, however, it is still connected 
since its associated undirected graph is the graph in Figure 29. 1a. 
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2. Forn > 1, let A” be the (n + 1) x (n+ 1) matrix be given by 


Coe ee 


0 

1 
AM = 1 
® Onxn 
1 

Then A? is nearly reducible. The digraph [(A“’) is called a rosette, and has the most arcs 
possible for a minimally connected digraph on n + 1 vertices. Suppose that n > 2, and let P = 


0 1 
f @ I,-2. Then 


Aa) 


is the decomposition for A” given in Fact 7 with m = 1. 


References 


[BG00] J. Bang-Jensen and G. Gutin. Digraphs: Theory, Algorithms and Applications. Springer-Verlag, 
London, 2000. 

[BH79] R.A. Brualdi and M.B. Hendrick. A unified treatment of nearly reducible and nearly decomposable 
matrices. Lin. Alg. Appl., 24 (1979) 51-73. 

[BJ86] W.W. Barrett and C.R. Johnson. Determinantal Formulae for Matrices with Sparse Inverses, II: 
Asymmetric Zero Patterns. Lin. Alg. Appl., 81 (1986) 237-261. 

[BP94] A. Berman and R.J. Plemmons. Nonnegative Matrices in the Mathematical Sciences. SIAM, Philadel- 
phia, 1994. 

[Bru82] R.A. Brualdi. Matrices, eigenvalues and directed graphs. Lin. Multilin. Alg. Applics., 8 (1982) 
143-165. 

[BR91] R.A. Brualdiand H.J. Ryser. Combinatorial Matrix Theory. Cambridge University Press, Cambridge, 
1991. 

[BR97] R.B. Bapat and T.E.S. Raghavan. Nonnegative Matrices and Applications. Cambridge University 
Press, Cambridge, 1997. 

[Cve75] D.M. Cvetkovic. The determinant concept defined by means of graph theory. Mat. Vesnik, 12 
(1975) 333-336. 

[FP69] M. Fiedler and V. Ptak. Cyclic products and an inequality for determinants. Czech Math J., 19 
(1969) 428-450. 

[Har62] E Harary. The determinant of the adjacency matrix of a graph. SIAM Rev., 4 (1962) 202- 
210. 

[LHE94] Z. Li, FE. Hall, and C. Eschenbach. On the period and base of a sign pattern matrix. Lin. Alg. Appl., 
212/213 (1994) 101-120. 

[Min88] H. Minc. Nonnegative Matrices. John Wiley & Sons, New York, 1988. 

[MOD89] J. Maybee, D. Olesky, and P. van den Driessche. Matrices, digraphs and determinants. SIAM J. 
Matrix Anal. Appl., 4, (1989) 500-519. 


29-14 Handbook of Linear Algebra 


[Rot75] U.G. Rothblum. Algebraic eigenspaces of nonnegative matrices. Lin. Alg. Appl., 12 (1975) 281-292. 

[Sch86] H. Schneider. The influence of the marked reduced graph of a nonnegative matrix on the Jordan 
form and related properties: a survey. Lin. Alg. Appl., 84 (1986) 161-189. 

[Stu91] J.L. Stuart. The determinant of a Hessenberg L-matrix, SIAM J. Matrix Anal., 12 (1991) 7- 
15. 

[Stu03] J.L. Stuart. Powers of ray pattern matrices. Conference proceedings of the SIAM Conference 
on Applied Linear Algebra, July 2003, at http://www.siam.org/meetings/la03/proceedings/stuartjl. 
pdf. 

[Var04] R.S. Varga. Gershgorin and His Circles. Springer, New York, 2004. 


30 


Bipartite Graphs and 
Matrices 


30.1 Basics of Bipartite Graphs ......................004. 30-1 

30.2 Bipartite Graphs Associated with Matrices .......... 30-4 
Bryan L. Shader 30.3 Factorizations and Bipartite Graphs................. 30-8 
University of Wyoming Referencesss oi Pia tb bbb Lad eb eee ddl dees 30-11 


An m xX n matrix is naturally associated with a bipartite graph, and the structure of the matrix is reflected 
by the combinatorial properties of the associated bipartite graph. This section discusses the fundamental 
structural theorems for matrices that arise from this association, and describes their implications for linear 
algebra. 


30.1 Basics of Bipartite Graphs 


This section introduces the various properties and families of bipartite graphs that have special significance 
for linear algebra. 


Definitions: 


A graph G is bipartite provided its vertices can be partitioned into disjoint subsets U and V such that 
each edge of G has the form {u,v}, where u € U and v € V. The set {U, V} is a bipartition of G. 

A complete bipartite graph is a simple bipartite graph with bipartition {U, V} such that each {u, v} 
(u € U,v € V) isan edge. The complete bipartite graph with |U| = m and |V| = n is denoted by Kn. 

A chordal graph is one in which every cycle of length 4 or more has a chord, that is, an edge joining 
two nonconsecutive vertices on the cycle. 

A chordal bipartite graph is a bipartite graph in which every cycle of length 6 or more has a chord. 

A bipartite graph is quadrangular provided it is simple and each pair of vertices with a common 
neighbor lies on a cycle of length 4. 

A weighted bipartite graph consists of a simple bipartite graph G anda function w : E — X, where E 
is the edge set of G and X isa set (usually Z, R, C, {—1, 1}, ora set of indeterminates). A signed bipartite 
graph is a weighted bipartite graph with X = {—1, 1}. Ina signed bipartite graph, the sign of a set a of 
edges, denoted sgn(q@), is the product of the weights of the edges in a. The set @ is positive or negative 
depending on whether sgn(q@) is +1 or —1. 

Let G be a bipartite graph with bipartition {U, V} and let 11, u2,..., Um, and v1, v2,..., V, be orderings 
of the distinct elements of U and V, respectively. The biadjacency matrix of G is the m x n matrix 
Bo = [bi], where bj is the multiplicity of the edge {u;, v;}. Note that if U, respectively, V, is empty, then 
Bg is a matrix with no rows, respectively, no columns. For a weighted bipartite graph, bj; is defined to be 
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the weight of the edge {u;, v;} if present, and 0 otherwise. For a signed bipartite graph, bj; is the sign of the 
edge {u;,v;} if present, and 0 otherwise. 

Let N¢ be an oriented incidence matrix of simple graph G. The cut space of G is the column space of 
Nz", and the cut lattice of G is the set of integer vectors in the cut space of G. The flow space of G is 
{x € R™ : NGx = 0}, and the flow lattice of G is {x € Z” : Nox = 0}. 

A matching of G is a set M of mutually disjoint edges. If M has k edges, then M is a k-matching, and 
if each vertex of G is in some (and hence exactly one) edge of M, then M is a perfect matching. 


Facts: 


Unless otherwise noted, the following can be found in [BR91, Chap. 3] or [Big93]. In the references, the 
results are stated and proven for simple graphs, but still hold true for graphs. 


1. A bipartite graph has no loops. It has more than one bipartition if and only if the graph is discon- 
nected. Each forest (and, hence, each tree and each path) is bipartite. The cycle C,, is bipartite if 
and only if n is even. 

2. The following statements are equivalent for a graph G: 


(a) G is bipartite. 


(b) The vertices of G can be labeled with the colors red and blue so that each edge of G has a red 
vertex and a blue vertex. 


(c) G has no cycles of odd length. 
(d) There exists a permutation matrix P such that P’ Ag P has the form 


O B 
Br oO} 


(e) G is loopless and every minor of the vertex-edge incidence matrix Ng of G is 0, 1, or —1. 


(f) The characteristic polynomial p4,(x) = >7jL9 c;x"? of Ag satisfies cy = 0 for each odd 
integer k. 


(g) o(G) = —o(G) (as multisets), where o(G) is the spectrum of Ag. 


3. The connected graph G is bipartite if and only if —p(Ag) is an eigenvalue of Ag. 
4. The bipartite graph G disconnected if and only if there exist permutation matrices P and Q such 
that PBg Q has the form 


B, O 
OB, 


> 


where both B, and By have at least one column or at least one row. More generally, if G is bipartite 
and has k connected components, then there exist permutation matrices P and Q such that 


B, O « O 
O By: O 

PBeQ=|,  . || 
Ov OF eso “By 


where the B; are the biadjacency matrices of the connected components of G. 

5. [GRO1] If G is a simple graph with n vertices, m edges, and c components, then its cut space has 
dimension n — c, and its flow space has dimension m — n + c. If G is a plane graph, then the edges 
of G can be oriented and ordered so that the flow space of G equals the cut space of its dual graph. 
The norm, x’ x, is even for each vector x in the cut lattice of G if and only if each vertex has even 
degree. The norm of each vector in the flow lattice of G is even if and only if G is bipartite. 


Bipartite Graphs and Matrices 30-3 


U, V; u, v, 
vy, Us Vy u, 
FIGURE 30.1 


6. [Bru66], [God85], [Sim89] Let G be a bipartite graph with a unique perfect matching. Then there 
exist permutation matrices P and Q such that PBg Q is a square, lower triangular matrix with all 
1s on its main diagonal. If G is a tree, then the inverse of PBg Q is a (0, 1, —1)-matrix. Let n be the 
order of Bg, and let H be the simple graph with vertices 1,2,... ,n and {i, j} an edge if and only 
ifi A j and either the (i, j)- or (j,i)-entry of PB Q is nonzero. If H is bipartite, then (PBg Q)~! 
is diagonally similar to a nonnegative matrix, which equals PBg Q if and only if G can be obtained 
by appending a pendant edge to each vertex of a bipartite graph. 


Examples: 


1. Up to matrix transposition and permutations of rows and columns, the biadjacency matrix of the 
path P,,, the path P2,41, and the cycle C2, are 


1 0 1 O 0 1 O 0 
0 1 0 1 0 0 1 1 0 
0 
0 0 1 1 0 0 1 1 0 0 0 1 1 
Oy ates 0 1 nxn, a a ae 0 1 nx(n+1), _ 0 1 nxn. 


2. The biadjacency matrix of the complete bipartite graph Kinny is Jina, the m x n matrix of all ones. 

3. Up to row and column permutations, the biadjacency matrix of the graph obtained from Ky, by 
removing the edges of a perfect matching is J, — In. 

4. Let G be the bipartite graph (Figure 30.1). 


Then 
100 0 
010 0 
Be= 
cara i es a 
0101 


G has a unique perfect matching, and the graph H defined in Fact 6 is the path 1-3—-2—4. Hence, 
Bg is diagonally similar to a nonnegative matrix. Also, since G is obtained from the bipartite graph 
vj —3—v,—1y by appending pendant vertices to each vertex, Bg' is diagonally similar to Be. 


Indeed, 
1 000 
SBo'S =S ae St=B 
Bes 8 Late <Siinar 0 Page 
0-1 01 


where S is the diagonal matrix with main diagonal (1, 1,—1, —1). 
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30.2 Bipartite Graphs Associated with Matrices 


This section presents some of the ways that matrices have been associated to bipartite graphs and surveys 
resulting consequences. 


Definitions: 


The bigraph of the m x n matrix A = [aj] is the simple graph with vertex set U U V, where U = 
{1,2,...,m} and V = {1’,2’,...,n"}, and edge set {{i,j’} : aj 4 0}. If A is a nonnegative 
integer matrix, then the multi-bigraph of A has vertex set U U V and edge {i, j’} of multiplicity aj. 
If A is a general matrix, then the weighted bigraph of A has vertex set U U V and edge {i, j’} of weight aj. 
If A is a real matrix, then the signed bigraph of A is obtained by weighting the edge {7, j’} of the bigraph 
by +1 if aj > 0, and by —1 ifaj < 0. 

The (zero) pattern of the m x n matrix A = [aj] is the m x n (0, 1)-matrix whose (i, j)-entry is 1 if 
and only if aj A 0. 

The sign pattern of the real m x n matrix A = [aj] is the m x n matrix whose (i, j )-entry is +, 0, or —, 
depending on whether aj; is positive, zero, or negative. (See Chapter 33 for more information on sign 
patterns.) 

A (0, 1)-matrix is a Petrie matrix provided the 1s in each of its columns occur in consecutive rows. A 
(0, 1)-matrix A has the consecutive ones property if there exists a permutation P such that P A isa Petrie 
matrix. 

The directed bigraph of the real m x n matrix A = [ajj] is the directed graph with vertices 1,2,... ,m, 
1’,2’,... ,n’, the arc (i, j") ifand only if aj > 0, and the arc (j’,i) if and only if aj < 0. 

An m xX n matrix A is a generic matrix with respect to the field F provided its nonzero elements are 
independent indeterminates over the field F. The matrix A can be viewed as a matrix whose elements are 
in the ring of polynomials in these indeterminates with coefficients in F. 

Let A be an n x n matrix with each diagonal entry nonzero. The bipartite fill-graph of A, denoted 
Gt (A), is the simple bipartite graph with vertex set {1,2,... ,n}U{1',2’,... ,n'} with an edge joiningi and 
j' ifand only if there exists a path from i to j in the digraph, (A), of A each of whose intermediate vertices 
has label less than min{i, j}. If A is symmetric, then (by identifying vertices i and i’ fori = 1,2,...,n 
and deleting loops), G*(A) can be viewed as a simple graph, and is called the fill-graph of A. 

The square matrix B has a perfect elimination ordering provided there exist permutation matrices P 
and Q such that the bipartite fill-graph, Gt (P BQ), and the bigraph of P BQ are the same. 

Associated with the n x n matrix A = [a,j] is the sequence Ho, H),... , H,—, of bipartite graphs as 
defined by: 


1. Hh consists of vertices 1,2,...,n, and 1’,2’,... ,n’, and edges of the form {i, j’}, where aj 4 0. 
2. Fork = 1,...,n—1, H; is the graph obtained from H;_, by deleting vertices k and k’ and inserting 
each edge of the form {r,c’}, where r > k,c > k, and both {r, k’} and {k, c’} are edges of Hy_1. 


The 4-cockades are the bipartite graphs recursively defined by: A 4-cycle is a 4-cockade, and if G is a 
4-cockade and e is an edge of G, then the graph obtained from G by identifying e with an edge of a 4-cycle 
disjoint from G is a 4-cockade. A signed 4-cockade is a 4-cockade whose edges are weighted by +1 in such 
a way that every 4-cycle is negative. 


Facts: 
General references for bipartite graphs associated with matrices are [BR91, Chap. 3] and [BS04]. 


1. [Rys69] (See also [BR91, p. 18].) If A is an m x n (0, 1)-matrix such that each entry of AA? is 
positive, then either A has a column with no zeros or the bigraph of A has a chordless cycle of 
length 6. The converse is not true. 

2. [RT76], [GZ98] If G = (V, E) is a connected quadrangular graph, then |E| < 2|V| — 4. The 
connected quadrangular graphs with | E | = 2|V| — 4 are characterized in the first reference. 
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3. [RT76], If Ais an m x n (0, 1)-matrix such that no entry of AA! or A’ Ais 1, and the bigraph of 


A is connected, then A has at most 2(m + n) — 4 nonzero entries. 


4, [Tuc70] The (0, 1)-matrix A has the consecutive ones property if and only if it does not have a 


10. 


submatrix whose rows and columns can be permuted to have one of the following forms for k > 1. 


1 0 --- 0 0 
1 1 0 0 141 0 1 
0 1 1 0 
0 1 
0 0 1 pt 
1 0 
ae 0 : (k+2) x (k+2), 
Oren OT) ects), 
1 0 0 0 1 
ae ett 1 10 0 0 0 1 10 0 0 
0 1 : 0 0 1 1 0 0 1 1 1 £41 £40 
11d , 0 001i 1 ; 0 1 1 £=+0 
0 0 110 OP oD Oe oh 8 Tgegs LE 0 Os ob, Eee 
0 men LT 43) (43) 


[ABH99] Let A be a (0, 1)-matrix and let L = D — AA’, where D is the diagonal matrix whose 
ith diagonal entry is the ith row sum of AA’. Then L is a symmetric, singular matrix each of 
whose eigenvalues is nonnegative. Let v be a eigenvector of L corresponding to the second smallest 
eigenvalue of L. If A has the consecutive ones property and the entries of v are distinct, then PA 
is a Petrie matrix, where P is the permutation matrix such that the entries of Pv are in increasing 
order. In addition, the reference gives a recursive method for finding a P such that P A is a Petrie 
matrix when the elements of v are not distinct. 

The directed bigraph of the real matrix A contains at most one of the arcs (i, j’) or (j’,i). 

[FG81] The directed bigraph of the real matrix A is strongly connected if and only if there do not 
exist subsets w and 6 such that A[a, B] > 0 and A(qa, B) < 0. Here, either a or 6B may be the empty 
set, and a vacuous matrix M satisfies both M > 0 and M < 0. 


. [FG81] If A = [aj] isa fully indecomposable, n x n sign pattern, then the following are equivalent: 


(a) There is a matrix A with sign pattern A such that A is invertible and its inverse is a positive 
matrix. 


(b) There do not exist subsets aw and f such that A[a, 8B] > O and A(a, B) < O. 

(c) The bipartite directed graph of A is strongly connected. 

(d) There exists a matrix with sign pattern A each of whose line sums is 0. 

(e) There exists a rank n — 1 matrix with sign pattern A each of whose line sums is 0. 

[Gol80] Up to relabeling of vertices, G is the fill-graph of some n x n symmetric matrix if and only 
if G is chordal. 

[GN93] Let A be an n x n (0, 1)-matrix with each diagonal entry equal to 1. Suppose that B is a 
matrix with zero pattern A, and that B can be factored as B = LU, where L = [€;,j] is a lower 
triangular matrix and U = [uj] isan upper triangular matrix. Ifi # j and either ¢ 4 Ooruy 4 0, 


then {i, j"} is an edge of G*(A). Moreover, if B is a generic matrix with zero pattern A, then such 
a factorization B = LU exists, and for each edge {i, j’} of G*(A) either i, A 0 or uj #0. 
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1 ' 2 3 
1 2 3 
FIGURE 30.2 


11. [GG78] If the bigraph of the generic, square matrix A is chordal bipartite, then A has a perfect 
elimination ordering and, hence, there exist permutation matrices P and Q such that performing 
Gaussian elimination on P AQ has no fill-in. The converse is not true; see Example 3. 

12. [GN93] If A is a generic n x n matrix with each diagonal entry nonzero, and aw = {1,2,...,r}, 
then the bigraph of the Schur complement of A[q@] in A is the bigraph H, defined above. 

13. [DG93] For each matrix with a given pattern, small relative perturbations in the nonzero entries 
cause only small relative perturbations in the singular values (independent of the values of the 
matrix entries) if and only if the bigraph of the pattern is a forest. The singular values of such a 
matrix can be computed to high relative accuracy. 

14. [DES99] If the signed bipartite graph of the real matrix is a signed 4-cockade, then small relative 
perturbations in the nonzero entries cause only small relative perturbations in the singular values 
(independent of the values of the matrix entries). The singular values of such a matrix can be 
computed to high relative accuracy. 


Examples: 
1. Let 


The directed bigraph of A is (Figure 30.2). 

Since this is strongly connected, Fact 7 implies that there do not exist subsets @ and 6 such 
that A[a, 8] > O and A(a, 8) < O. Also, there is a matrix with sign pattern A whose inverse is 
positive. One such matrix is 


any ae 
i 2" 
1 0 =2 


2. A signed 4-cockade on 8 vertices (unlabeled edges have sign +1) and its biadjacency matrix are 
(Figure 30.3) 


1! 
1 1 0 0 
1 —-1 1 1 
0 1 1 0 
0 1 0 1 


FIGURE 30.3 
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3. Both the bipartite fill-graph and the bigraph of the matrix (Figure 30.4) below 


- Oo fF CF CO 
Co FF OF CO 
pe PF OF Oo CO 
=) 
ee 
re OO Oo COC CUD 


FIGURE 30.4 


are the graph illustrated. Since its bigraph has a chordless 6-cycle, this example shows that the 
converse to Fact 11 is false. 


4, Let 
x1 X%2 X30 Xa 
= Xs = (X6 0 0 
x7 O xg O 
X9 0 0 X10 
where xX),... 5X19 are independent indeterminates. The bigraph of A is chordal bipartite. The 


biadjacency matrix of Hj is J3. Thus, by Fact 12, the pattern of the Schur complement of A[{1}] in 
Ais J3. The bipartite fill-graph of A has biadjacency matrix J4. If 


re CO CO CO 
> ei OS 2 
oor f& 
Or 3 a. 


then the bipartite fill-graph of PAP’ and the bigraph of PAP’ are the same. Hence, it is possible to 
perform Gaussian elimination (without pivoting) on PAP? without any fill-in. 


Applications: 


1. [Ken69], [ABH99] Petrie matrices are named after the archaeologist Flinders Petrie and were first 
introduced in the study of seriation, that is, the chronological ordering of archaeological sites. If 
the rows of the matrix A represent archaeological sites ordered by their historical time period, the 
columns of A represent artifacts, and aj = 1 if and only if artifact j is present at site 1, then one 
would expect A to be a Petrie matrix. More recently, matrices with the consecutive ones property 
have arisen in genome sequencing (see [ABH99]). 

2. [BBS91], [Sha97] If U is a unitary matrix and A is the pattern of U, then the bigraph of A is 
quadrangular. If U is fully indecomposable, then U has at most 4n — 4 nonzero entries. The matrices 
achieving equality are characterized in the first reference. (See Fact 2 for more on quadrangular 
graphs.) 
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30.3 Factorizations and Bipartite Graphs 


This section discusses the combinatorial interpretations and applications of certain matrix factorizations. 


Definitions: 


A biclique of a graph G is a subgraph that is a complete bipartite graph. For disjoint subsets X and Y of 
vertices, B(X, Y) denotes the biclique consisting of all edges of the form {x, y} such that x € X andy € Y 
(each of multiplicity 1). If G is bipartite with bipartition {U, V}, then it is customary to take X C U and 
YCYV. 

A biclique partition of G = (V, E) is a collection B(X, Y;),... , B(Xx, Yx) of bicliques of G whose 
edges partition E. 

A biclique cover of G = (V, E) is a collection of bicliques such that each edge of E is in at least one 
biclique. 

The biclique partition number of G, denoted bp(G), is the smallest k such that there is a partition 
of G into k bicliques. The biclique cover number of G, denoted bc(G), is the smallest k such that there 
is a cover of G by k bicliques. If G does not have a biclique partition, respectively, cover, then bp(G), 
respectively, bc(G), is defined to be infinite. 

If G is a graph, then n,(G), respectively, n_(G), denotes the number of positive, respectively, negative, 
eigenvalues of Ag (including multiplicity). 


If X C {1,2,... ,n}, then the characteristic vector of X is the n x 1 vector X = [x;], where x; = 1 if 
i € X,and x; = 0 otherwise. 

The nonnegative integer rank of the nonnegative integer matrix A is the minimum k such that there 
exist an m x k nonnegative integer matrix B andak x n nonnegative integer matrix C with A = BC. 

The (0,1)-Boolean algebra consists of the elements 0 and 1, endowed with the operations defined by 
0+0=0,04+1=1=14+0,1+1=1,0*1=0=1+*0,0+*0=0,and 1+ 1=1.A Boolean matrix 
is a matrix whose entries belong to the (0,1)-Boolean algebra. Addition and multiplication of Boolean 
matrices is defined as usual, except Boolean arithmetic is used. 

The Boolean rank of the m x n Boolean matrix A is the minimum k such that there exists an m x k 
Boolean matrix B andak x n Boolean matrix C such that A = BC. 

Let G be a bipartite graph with bipartition {{1,2,... , m}, {1',2',... ,n'}}. Then M(G) denotes the set 
of all m x n matrices A = [aj] such that if aj 4 0, then {i, j’} is an edge of G, that is, the bigraph of A 
is a subgraph of G. The graph G supports rank decompositions provided each matrix A € M(G) is the 
sum of rank(A) elements of M(G) each having rank 1. 

If G isa signed bipartite graph, then M(G) denotes the set of all matrices A = [aj] such that ifaj > 0, 
then {i, j’} is a positive edge of G, and if aj < 0, then {i, j"} is a negative edge of G. The signed bigraph 
G supports rank decompositions provided each matrix A € M(G) is the sum of rank(A) elements of 
M(G) each having rank 1. 


Facts: 
1. [GP71] 
* A graph has a biclique partition (and, hence, cover) if and only if it has no loops. 
* For every graph G, bc(G) < bp(G). 
* Every simple graph G with n vertices has a biclique partition with at most n — 1 bicliques, namely, 
B({i}, {7 : {i j} is an edge of G and j > i}) G@ = 1,2,...,n—1). 


2. [CG87] Let G be a bipartite graph with bipartition (U, V), where |U| = m and |V| = n. Let 
B(X), Y), B(X2, Y2),... 5 BUX; Ye) be bicliques with X; C U and Y; C V for alli. The following 
are equivalent: 


(a) B(X,, Y\), B(X2, Y2),..., B(Xx, Ye) is a biclique partition of G. 
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10. 


11. 


> oT 


(b) SEXY, =Be. 


(c) XY? = Bg, where X is the n x k matrix whose ith column is X;, and Y is the n x k matrix 


ok 
whose ith column is Y;. 


[CG87] For a simple bipartite graph G, bp(G) equals the nonnegative integer rank of Bg. 
[CG87] 


* Let G be the bipartite graph obtained from K,,,, by removing a perfect matching. Then bp(G) = 
n. Furthermore, if B(X;, Y;) (i = 1,2,... ,m) isa biclique partition of G, then there exist positive 
integers r and s such that rs = n — 1, |X;| =r and |Y;| =s (i = 1,2,... ,n), k is in exactly r 
of the X;’s and exactly s of the Y;’s (k = 1,2,... ,n), and X; 1 Y; = 1 fori F j. 

¢ In matrix terminology, if X and Y aren x n (0, 1)-matrices such that X y! = J, —I,,, then there 


exist integers r ands such that rs = n— 1, X has constant line sums, Y has constant line sums 
s,and Y7X = J, — Ih. 


In particular, if n — 1 is prime, then either X is a permutation matrix and Y = (J, — I,)X, or 
Y is a permutation matrix and X = (J, — I,)Y. 


[BS04, see p. 67] Let G be a graph on n vertices with adjacency matrix Ag, and let B(X), Yj), 
B(X, Yo), ... 5 B(Xx, Yx) be bicliques of G. Then the following are equivalent: 


(a) B(X,, Y,), B(X2, Y2),..., B(Xx, Yx) is a biclique partition of G. 


eo5T 


> oT 
(b) SX, + es Y¥iXi = Ag. 


(c) XY? + YX? = Ag, where X is the n x k matrix whose ith column is Ke and Y is then x k 


Ss 
matrix whose ith column is Yj. 


(d) Ag =M 2 M", where M is the n x 2k matrix [x y| formed from the matrices X 
and Y defined in (c). 


[CH89] The bicliques B(X), Y|), B(X2, Y2),..., BCX; Ye) partition K,, if and only if XY? isan 
n X n tournament matrix, where X is the n x k matrix whose ith column is X,, and Y isthen x k 
matrix whose ith column is Y,. Thus, bp(K,,) is the minimum nonnegative integer rank among all 
the n x n tournament matrices. 


[CH89] The rank of an n x n tournament matrix is at least n — 1. 
(Attributed to Witsenhausen in [GP71]) 


bp(K,) =n-1, 


that is, it is impossible to partition the complete graph into n — 2 or fewer bicliques. 
[GP71] The Graham—Pollak Theorem: If G is a loopless graph, then 


bp(G) = max{n;(G),n_(G)}. (30.1) 


The graph G is eigensharp if equality holds in (30.1). It is conjectured in [CGP86] that for all 4, 
and n sufficiently large, the complete graph AK, with each edge of multiplicity k is eigensharp. 
[ABS91] If B(X1, Y1), B(X2, Y2), ... , B(Xx, Yx) is a biclique partition of G, then there exists an 
acyclic subgraph of G with max{(n+(G),n_(G)} edges no two in the same B(X;, Y;). 

In particular, for each biclique partition of K,, there exists a spanning tree no two of whose edges 
belong to the same biclique of the partition. 

[CH89] For all positive integers r and s with 2 < r < s, the edges of the complete graph K>,. 
cannot be partitioned into copies of the complete bipartite graph K,.. 
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12. 


13. 


14. 


15. 


16. 


17. 
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[Hof01] If m and n are positive integers with 2m < n, and G,,, is the graph obtained from 
the complete graph K,, by duplicating the edges of an m-matching, then n4(G) = n-—m—1, 
n_(G) =m+1,andbp(G) >n-—m+ |/2m| — 1. 
[CGP86] Let A be an m x n (0, 1)-matrix with bigraph G and let B(X,,Y), B(X2, Yo), ... 
B( Xx, Y;) be bicliques. The following are equivalent: 


(a) B(X,, Y,), B(X2, Y2),..., BCX, Yx) is a biclique cover of G. 
> —T 
(b) yey XiY; = A (using Boolean arithmetic). 
(c) XY? = B (using Boolean arithmetic), where X is the m x k matrix whose jth column is Xx. 


Ee 
and Y is the m x k matrix whose j column is Yj. 


[CGP86] The Boolean rank of a (0,1)-matrix A equals the biclique cover number of its 
bigraph. 

[CSS87] Let k bea positive integer and let t(k) be the largest integer n such that there exists ann x n 
tournament matrix with Boolean rank k. Then for k > 2, t(k) < k!°®?), and n(n? +n+1)+2 < 
t(n?+nt+1). 

It is still an open problem to determine the minimum Boolean rank among n x n tournament 
matrices. 

[DHM95, JM97] The bipartite graph G supports rank decompositions if and only if G is chordal 
bipartite. 

[GMS96] The signed bipartite graph G support rank decompositions if and only if 


sen(y) = (1) (30.2) 


for every cycle y of G of length €(y) > 6. Additionally, every matrix in M(G) has its rank equal 
to its term rank if and only if (30.2) holds for every cycle of G. 


Examples: 


1. 


2 


Below, the edges of different textures form the bicliques (Figure 30.5) in a biclique partition of the 
graph G24 obtained from K, by duplicating two disjoint edges. 

Let 1 be an integer and r and s positive integers with n — 1 = rs. Then X y? = J, — I,, where 
X=1I4+C84+C% +---4+-06-) yY=C+C?+C34---+C%, and C is then x n permutation 
matrix with 1s in positions (1,2), (2,3),...,(m—1,n), and (n, 1). 

This shows that for each pair of positive integers r ands withrs = n—1, there isa biclique partition 
of J, — I, with X; and Y; satisfying the conditions in Fact 4. 

For n odd, B({i}, {i + 1,i+2,...,i+ n1}) (i = 1,2,... ,n) isa partition of K,, into bicliques 
each isomorphic to K Ltsh> where the indices are read mod n (see Fact 11). 


FIGURE 30.5 
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The permanent is a matrix function introduced (independently) by Cauchy and Binet in 1812. At first sight 
it seems to be a simplified version of the determinant, but this impression is misleading. In some important 
respects the permanent is much less tractable than the determinant. Nonetheless, permanents have found 
a wide range of applications from pure combinatorics (e.g., counting problems involving permutations) 
right through to applied science (e.g., modeling subatomic particles). For further reading see [Min78], 
[Min83], [Min87], [CW05], and the references therein. 


31.1 Basic Concepts 


Definitions: 


Let A = [a,j] be an m x n matrix over a commutative ring, m < n. Let S be the set of all injective functions 
from {1,2,...,m} to {1,2,...,m} (in particular, ifm = n, then S is the symmetric group on {1,2,...,n}). 
The permanent of A is defined by 


per(A) = x Il Qio(i)- 


o€S i=1 


Two matrices A and B are permutation equivalent if there exist permutation matrices P and Q such that 
B=PAQ. 
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Facts: 
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For facts for which no specific reference is given and for background reading on the material in this 
subsection, see [Min78]. 


1. If A is any square matrix, then per(A) = per(A7). 


10. 


11. 


12. 


13. 


Our definition implies that per(A) = 0 for all m x n matrices A, where m > n. Some authors 
prefer to define per(A) = per(A‘) in this case. 


. If Ais any m x n matrix and P and Q are permutation matrices of respective orders m and n, then 


per(P AQ) = per(A). That is, the permanent is invariant under permutation equivalence. 

If A is any m x n matrix and c is any scalar, then per(c A) = c™” per(A). 

The permanent is a multilinear function of the rows. If m = n, it is also a multilinear function of 
the columns. 

It is not in general true that per(AB) = per(A) per(B). 


A 


If M has block decomposition M = B , where either A or C is square, then per(M) = 


per(A) per(C). 

Let A be an m x n matrix with m < n. Then per(A) = Oif A contains ans x (n—s +1) submatrix 
of zeroes, for somes € {1,2,..., mh}. 

(Laplace expansion) If Ais an m x n matrix, 2 < m < n,anda@ € Q,m,where1 <r <m < nthen 


per(A) = Se per (Ala, B]) per (A(q, B)). 
BEQrn 


In particular, for anyi € {1,2,...,m}, 
per(A) = s aj per (A(i, j)). 
If A and B aren x n matrices and s and ¢ are arbitrary scalars, then 


per(s A + tB) => pt ye per (Ala, B]) per (B(a, B)) 


a, BEQrn 


(where we interpret the permanent of a 0 x 0 matrix to be 1). 
(Binet-Cauchy) If A and B are m x n and n x m matrices, respectively, where m < n, then 


1 
per(AB) = ye ial per (A[{1,2,...,m},a]) per (Bla, {1,2,...,m}]). 
a (a 
The sum is over all nondecreasing sequences a of m integers chosen from the set {1,2,...,} and 
[A(a) = a! a! ---a@,!, where a; denotes the number of occurrences of the integer 7 in a. 


[MM62], [Bot67] Let F bea field and m > 3 an integer. Let T be a linear transformation for which 
per (T(X )) = per(X) for all X ¢ F”*". Then there exist permutation matrices P and Q and 
diagonal matrices D,; and D) such that per(D; D2) = 1 and either T(X) = D,P X QD), for all 
X € F”*" or T(X) = D, PX" QD) forall X € F™*”. 

(Alexandrov’s inequality) Let A = [aj] € R’“" andl <r <s <n. Ifaj > 0 whenever j ¥# s, 
then 


(per(A a> Ya per(A(i, s)) a per(A(i,r)). (31.1) 


Moreover, if aj; > 0 whenever j ¥ s, then equality holds in (31.1) iff there exists c € R such that 
air = caj;; for alli. 
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14. If G is a balanced bipartite graph (meaning the two parts have equal size), then per(Bg) counts 
perfect matchings (also known as 1-factors) in G. 

15. If Disa directed graph, then per(Ap) counts the cycle covers of D. A cycle cover is a set of disjoint 
cycles which include every vertex exactly once. 


Examples: 
1. 
koe 5 6 4 6 4 5 
per|4 5 6] =1 per + 2 per + 3 per 
8 9 i, 7 8 
7 8 9 
=1-5-94+1-6-84+2-4-9+2-6-74+3-4-84+3-5-7= 450. 
023 +4 
2. per|}5 6 O 8] =2-5-1242-8-943-5-12+3-6-9+3-6-12+3-8:9+4-6-9 = 1254. 
9 0 0 12 
2 3 =4 5-2 =o “Sif 
3. If A = ; a and B = | 1 i AB = & 5 . Hence, 80 = per(AB) 4 


per(A) per(B) =2x2=4. 
4. Below is a bipartite graph G (Figure 31.1) and its biadjacency matrix Bg. 


1 1 0 0 
0 0 1 0 
1 10 1 
FIGURE 31.1 0011 


Now, per(Bg) = 2, which means that G has two perfect matchings (Figure 31.2 and Figure 31.3). 


Lek eu) 


FIGURE 31.2 FIGURE 31.3 


5. The matrix in the previous example can also be interpretted as Ap for the directed graph D 
(Figure 31.4). It has two cycle covers (Figure 31.5 and Figure 31.6). 


2 Q 2 2 
1 1 1 
) e J ve) 
3 4 3 4 3 4 


FIGURE 31.4 FIGURE 31.5 FIGURE 31.6 


31.2 Doubly Stochastic Matrices 


Facts: 


For facts for which no specific reference is given and for background reading on the material in this 
subsection, see [Min78]. 


1. If A € Q,, then per(A) < 1 with equality iff A is a permutation matrix. 
2. [Ego81], [Fal81] If A € Q, and A # Jn then per(A) > per(4J,) =n!/n". 
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3. [Fri82] If A € Q, and Ahasa p x q submatrix of zeros (where p + q <n), then 


(n— p)! (n—@q)! (n—p-—q)"?4 
A 
per(A) > Gao) ag espa) 


Equality is achieved by any matrix permutation equivalent to the matrix A = [aj] defined by 


0 ifi < pandj <q, 
er ifi < pandj >q, 
ay = 
= ifi > pandj <q, 
Gone a ifi > pandj > q. 


If p + q #n-— 1, then no other matrix achieves equality. 

4. [BN66] If A is any n x n row substochastic matrix and 1 <r < n, then per(A) < m,, where m, is 
the maximum permanent over all r x r submatrices of A. 

5. [Min78, p. 41] If A = [aj] is a fully indecomposable matrix, then the matrix S = [sj] defined by 
Sij = aj per (A(i, j )) / per(A) is doubly stochastic and has the same zero pattern as A. 


Examples: 


1. The minimum value of the permanent in Qs is 24/625, which is (uniquely) achieved by 


1/5 1/5 1/5 1/5 1/5 
1/5 1/5 1/5 1/5 1/5 
tJs= {1/5 1/5 1/5 1/5 1/5 
1/5 1/5 1/5 1/5 1/5 
1/5 1/5 1/5 1/5 1/5 
2. Inthe previous example, if we require that two specified entries in the same row must be zero, then 
the minimum value that the permanent can take is 1/24, which is (uniquely, up to permutation 
equivalence) achieved by 
0 oO 1/3 1/3 1/3 
1/4 1/4 1/6 1/6 1/6 
1/4 1/4 1/6 1/6 1/6]. 
1/4 1/4 1/6 1/6 1/6 
1/4 1/4 1/6 1/6 1/6 


3. A nonnegative matrix of order n > 3 can have zero permanent even if we insist that each row and 
column sum is at least one. For example, 


1 1 0 
perjO O 1) =0. 
0 0 1 


4. Suppose n identical balls are placed, one ball per bucket, in n labeled buckets on the back of a truck. 
When the truck goes over a bump the balls are flung into the air, but then fall back into the buckets. 
Suppose that the probability that the ball from bucket i lands in bucket j is pj. Then the matrix 
P = [pj] is row stochastic and per(P) is the probability that we end up with one ball in each 
bucket. That is, per(P) is the permanence of the initial state. 
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31.3. Binary Matrices 


Definitions: 


A binary matrix is a matrix in which each entry is either 0 or 1. 
AK is the set of n x n binary matrices in which each row and column sum is k. 
For an m x n binary matrix M the complement M“ is defined by M° = Jn, — M. 


A system of distinct representatives (SDR) for the finite sets S,, S,,..., S, is a choice of x), X2,...,Xn 
with the properties that x; € S; for each i and x; 4 x; whenever i $ j. 
The incidence matrix for subsets S;, S2,..., Sj, of a finite set {x), x2,...,X,} is the m x n binary matrix 


M = [mj] in which mj = 1 iff x; € Sj. 
P(2..-n) is the permutation matrix for the full cycle permutation (12---1). 


Facts: 


For facts for which no specific reference is given and for background reading on the material in this section, 
see [Min78]. 


1. The number of SDRs for a set of sets with incidence matrix M is per(M). 

2. If Me A‘, then ;M € Q,, so the results of the previous subsection apply. 

3. [Min78, p. 52] Let A be an m x n binary matrix where m < n. Suppose A has at least t positive 
entries in each row. Ift < mand per(A) > 0,then per(A) > t!.Ift > m, then per(A) > ft!/(t—m)!. 


4. IfAeE pat then there exist permutation matrices P}, P),..., P, such that 
k 
A= SSP, 
i=l 
5. [Bré73], [Sch78] Let A be any 1 x n binary matrix with row sums 1), r2,...5fn. Then 
per(A) < | [(ri!)!/" (31.2) 


i=l 

with equality iff A is permutation equivalent to a direct sum of square matrices each of which 

contains only 1s. 
6. [MW98] If m > 5, then (J; ® Jk ®--- @ Jx)* (where there are m copies of J,) maximizes the 
permanent in Ae The result is not true for m = 3. 
7. [Wan99b] For each k > 1 there exists N such that for all n > N a matrix M maximizes the 
permanent in Ak iff M ® J, maximizes the permanent in A‘ ,,. 
8. [Wan03] Ifn = tk+r withO <r <k<n,thenk!'r! < max per(A) < (kiy"/*, 


1/n 
9. [Wan03] Ifk = o(n) as n — oo, then ( max per(A)) ~ (kIyVE, 
Achy 


10. [GM90] Suppose 0 < k = O(n!~*) fora constant § > 0 as n — oo. Then 


n—kyn k 3h—k 2k —-k 
per(A) = n!(——) exp + 
n 


2n 6n2 4n3 


60n* n* n> n> 


15k4 + 70k3 — 105k? +32k  z  2z(2k—1) (s )) 
+— + +O 
for all A € nek , where z denotes the number of 2 x 2 submatrices of A that contain only zeros. 
In particular, if 0 < k = O(n'~*) for a constant 6 > 0 as n — oo, then per(A) is asymptotically 
equal to n!(1 — k/n)" for all A € Ao 
11. [Wan06] If2 < k < nasn > o, then 
Yn (k= 1)k! 


( min per(A)) (k=2 


AeA 
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12. [BN65] For any integers k > 1 andn > log, k + 1 there exists a binary matrix A of order n such 
that per(A) = k. 

13. The permanent of a square binary matrix counts permutations with restricted positions. This 
means that for each point being permuted there is some set of allowable images, while other images 


are forbidden. 


Examples: 


1. If D, = If, then 


11 “Al 
per(D,) = ni(1 it oi +++ (-1) =) 


is the number of derangements of n things, that is, the number of permutations of n points that 
leave no point in its original place. The 4"" derangement number is 


01 i141 

1 1 1 
per(D4) = per es =9, 

1 110 


2. The number of ways that n married couples can sit around a circular table with men and women 
alternating and so that nobody sits next to their spouse is 2 n! M,,, where M,, is known as the n-th 
menagé number and is given by 


a 2 2n — 
Mn = per (In + Poi..ny)°) = Ss 1)" n : ( " ") (n—r)l. 


r=0 
00111 
100 1 1 
The 5 menagé number is M5; = per|1 1 0 0 1] =13. 
1 110 0 
0 1 i1éi1é+%0 


3. SDRs are important for many combinatorial problems. For example, ak x n Latin rectangle (where 
k <n) isak x n matrix in which n symbols occur in such a way that each symbol occurs exactly 
once in each row and at most once in each column. The number of extensions of a given k x n 
Latin rectangle R toa (k + 1) x n Latin rectangle is the number of SDRs of the sets $1, S2,...5 Sy 
defined so that S; consists of the symbols not yet used in column i of R. 

4. [CW05] For n < 11 the minimum values of per(A) for A € ye are as follows: 


n=2 7 8 9 10 ll 


Ne 
NN KF |W 


12 «17 24 33 42 60 83 

oa — 24 44 80 144 248 440 764 1316 
— 120 265 578 1249 2681 5713 12105 
12000 30240 75510 
5040 14833 43386 126117 364503 
40320 133496 439792 1441788 
362880 1334961 4890740 
3628800 14684570 
39916800 


Moe ernaueb wnee 

| 

| 

| 

| 

NI 
N 
S 
e 
0 
on 
nS 
ow 
NX 
w 
oo 
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5. [MW98] For n < 11 the maximum values of per(A) for A € Ak are as follows: 


k n=2 3 4 5 6 7 8 9 10 11 

1 1 1 1 1 1 1 1 1 

2 2 2 =4 4 8 8 16 16 32 32 

3 - 6 9 13 36 54 81 216 324 486 

4 - — 24 44 82 148 576 1056 1968 3608 

5 — — — 120 265 580 = 1313 2916 14400 31800 

6 _ —- — — 720 1854 4752 12108 32826 86400 

7 5040 14833 43424 127044 373208 
8 40320 133496 440192 1448640 
9 362880 1334961 4893072 
10 3628800 14684570 
ll 39916800 


31.4 Nonnegative Matrices 


Definitions: 
Ak is the set of n x n matrices of nonnegative integers in which each row and column sum is k. 


Facts: 
1. [Min78, p. 33] Let A be an m x n nonnegative matrix with m < n. Then per(A) = 0 iff A contains 


ans xX (n—s + 1) submatrix of zeros, for somes € {1,2,...,m}. 

2. [Min78, p. 38] Let A be a nonnegative matrix of order n > 2. Then A is fully indecomposable iff 
per (A(i, j)) > 0 for all i, j. 

(k = 1)k-! )' 


kk-2 


2n 


< jae per(A) < aN 


4. [Min83] It is conjectured that min per(A) = min per(A). 

AeA AeA 
5. [Sou03] Let P denote the gamma function and let A be a nonnegative matrix of order n. In rowi 
of A, let m; and r; denote, respectively, the largest entry and the total of the entries. Then, 


n m; [Ti 
per(A) < [[@ (r (4 + ‘) ; (31.3) 


6. [Min78, p. 62] Let A be a nonnegative matrix of order n. Define s; to be the sum of the 7 smallest 
entries in row i of A. Similarly, define 5; to be the sum of the 7 largest entries in rowi of A. Then 


IIs: < per(A) < Il Sj. 
i=l i=l 


3. [Sch98] ( 


7. [Gib72] If A is nonnegative and z is a root of per(zI — A), then || < p(A). 


Examples: 
1. [BB67] If A is row substochastic, the roots of per(zJ — A) = 0 satisfy |z| < 1. 
2. If Soules’ bound (31.3) is applied to matrices in Ak it reduces to Brégman’s bound (31.2). 
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31.5 (+1)-Matrices 


Facts: 

1. [KS83] If A is a (£1)-matrix of order n, then per(A) is divisible by 2”~"o#("+)1, 

2. [Wan74] If H isan n x n Hadamard matrix, then | per(H)| < | det(H)| = n"/?. 

3. [KS83], [Wan05] There is no solution to| per(A)| = | det(A)| among the nonsingular (+1)-matrices 
of order n when n € {2,3,4} or n = 2 — 1 for k > 2, but there are solutions when n € 
{5,...,20} \ {7,15}. 

4, [Wan05] There exists a (+1)-matrix A of order n satisfying per(A) = 0 iffn+ 1 is not a power of 2. 


Examples: 
1. The 11 x 11 matrix A = [aj] defined by 


—1 if j —i € {1,2,3,5,7, 10}, 
aig = 
J +1 otherwise, 


satisfies per(A) = 0. No smaller (+1)-matrix of order n = 3 mod 4 has this property. 
2. The following matrix has per = det = 16, and is the smallest example (excluding the trivial case 
of order 1) for which per = det among +1-matrices. 


Sey) SE. wa; aT Se. 
Sos er ap ee a 
a eee ee ee 
eT el, SEL oh ad 
ao SH. sh Tet 


31.6 Matrices over C 


Facts: 
1. If A= [ay] € C”*" and B = [by] € R"™*” satisfy bi = |ajj|, then | per(A)| < per(B). 
2. If Ae C”*", then per(A) = per(A) = per(A*). 
3. [Min78, p. 113] If A € C”*” is normal with eigenvalues Aj, A2,...5An, then 


1 n 
| per(A)| < = Se ie 
i=l 
4. [Min78, p. 115] Let A € C"*” and let Aj,...,A, be the eigenvalues of AA*. Then 


1 n 
A)? <—S oat. 
| per(A)| oe ; 


5. [Lie66] If A= a € PD,, then per(A) > per(B) per(D). 
6. [JP87] Suppose a C B C {1,2,...,n}. Then for any A € PD,, 


det(A[B, B]) per(A(, B)) < det(A[a, a]) per(A(a, @)). 


(We interpret det or per of a0 x 0 matrix to be 1.) 
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7. [MN62] If A € C”*”" and B € C"*”, then 
| per(AB)|* < per(AA*) per(B*B). 


8. [Bre59] If A = [ay] € C"*” satisfies |a;;| > >> 4; |aij| for each i, then per(A) # 0. 


j#i 


Examples: 
1. If U is a unitary matrix, then | per(U)| < 1. 


31.7 Subpermanents 


Definitions: 


The k-th subpermanent sum, per, (A) of an m x n matrix A, is defined to be the sum of the permanents 
of all order k submatrices of A. That is, 


per,(A) = }> per(Ala, B]). 
pee 


By convention, we define pery(A) = 1. 


Facts: 


For facts for which no specific reference is given and for background reading on the material in this 
subsection see [Min78]. 


For each k, per, is invariant under permutation equivalence and transposition. 

If A is any m x n matrix and c is any scalar, then per,(c A) = ckper, (A). 

[BN66] If A is any n x n row substochastic matrix and 1 <r <n, then per,(A) < (”). 
[Fri82] per,(A) > per, (4Jn) for every A € Q,, and integer k. 


BPs ROT ES 


k} 
5. [Wan03] If A € AK andi < k, then per;(A) > (") opr 
i —i)! 


6. [Wan03] For 1 <i,k < nandAeé Mi 


(‘") hates o(@ a) SpaA)-< tl 
1 1—2 1 


7. [Wan03] For A € A‘, let & = (per;(A)/("))/". Then 


(k —1)*} 
ee ee 


8. [Nij76], [HLP52, p. 104] Let A be a nonnegative m x n matrix with per(A) 4 0. Forl <i < m—1, 


per; (A) ey (i + 1)(m —i + 1) per;,,(A) 7 per;,,(A) 
per;_,(A) — i(m —i) per; (A) per,(A) © 


9. [Wan99b] For A ¢ Ak, 


per; (A) S it+l1 
per;,,(A) ~ (ni)? 
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10. [Wan99a] Let k > 0 be an integer. There exists no polynomial p;(1) such that for alln and A € A3, 


Pet, —k-1 (A) 


pe A) < px(n). 


11. If G isa bipartite graph, then per,(6q) counts the k-matchings in G. A k-matching in G isa set of 
k edges in G such that no two edges share a vertex. 


Examples: 
1. For any matrix A the sum of the entries in A is per, (A). 
2. Any m x n matrix A has per,,(A) = per(A). 
3. perpJn) = ki(R)’. 
4. [Wan99a] Let A € AK have s submatrices which are copies of J. Then 
* per,(A) = kn. 
* per,(A) = skn(kn —2k+1). 
* per;(A) = ikn(k?n? — 6k?n + 3kn + 10k? — 12k + 4). 
* per,(A) = tkn(ken? — 12k n? + 6k?n? + 52k?n — 60k?n + 19kn — 84k? 
+ 168k? — 120k + 30) +s. 


* per;(A) = Zokn[kén* — 20k*n? + 10k?n? + 160k*n? — 180k? n? + 55k? n? 
— 620k*n + 1180k?n — 800k?n + 190kn + 1008k* — 2880k? 


+ 3240k? — 1680k + 336] + (nk — 8k + 8)s. 


5. The subpermanent sums are also known as rook numbers since they count the number of place- 
ments of rooks in mutually nonattacking positions on a chessboard. Let A be a binary matrix in 
which each 1 denotes a permitted position on the board and each 0 denotes a forbidden position for 
a rook. Then per; (A) is the number of placements of i rooks on permitted squares so that no two 
rooks occupy the same row or column. For example, the number of ways of putting 4 nonattacking 
rooks on the white squares of a standard chessboard is 


104103101 +0 
01010101 
101010 1 0 
010310101 
per, = 8304. 
1010101 0 
01010101 
1010101 0 
01010101 


31.8 Rook Polynomials 


Definitions: 


Let A be an m x n binary matrix. The polynomials p,(A,x) = S7y15 per; (A)x! and p2(A,x) = 
So g(—1)/per;(A)x"~ are both called rook polynomials because they are generating functions for the 
rook numbers. 


Let £;(x) be the k™ Laguerre polynomial, normalized to be monic. That is, 


k k _y\i 
x(x) = (—1)Fkt > (7) * 


i=0 
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Facts: 


For facts for which no specific reference is given and for background reading on the material in this section, 
see [Rio58]. 


ee tS 


When A is an m x n binary matrix, p;(A,x) = (—x)"p2(A, 1), 


x 


If A= B @C, then p;(A,x) = p(B, x)p,(C, x) and p2(A, x) = p2(B, x) p2(C, x). 

[HL72], [Nij76] For any nonnegative matrix A, all the roots of p;(A, x) and p2(A, x) are real. 
[HL72] For2<k<nandAe ANS the roots of 0;(A, x) are less than —1/(4k — 4), while the 
roots of 02(A, x) lie in the interval (0, 4k — 4). 

[JR80], [God81] For a square binary matrix A, with complement A‘, 


lo-e) 
per(A) = ' e ~*~ 07(A‘, x) dx. 
0 
[God81] For ann x n binary matrix M, 
p2(M, x) = S> per;(M")e,,-i(x). 
i=0 


[Sze75, p. 100] For any j,k let 5; denote the Kronecker delta. Then 


| e*6;(x)ee(x) dx = 8) 4k!. 
0 


Examples: 


1. 
2. 
os 


pi(P,x) = (x +1)" and p2(P,x) = (x — 1)" for a permutation matrix P € A}. 
P2r(Tk, x) = ly (x). 
Let C,, = I, + P(iz..n). Then 


pi(Cnx) = >> al ce 


and p2(Cy, 4x2) = 2Ton(x), where T;,(x) is the n™ Chebyshev polynomial of the first kind. 
pr(15,x) = Doyo (7) 4i(x) for any integer n > 1. 

The ideas of this chapter allow a quick computation of the permanent of many matrices that are 
built from blocks of ones by recursive use of direct sums and complementation. For example, 


per (Je Hi,.)") = f 
is k41 

=[ etx) So & ts) = ea k= (k+ DIKI, 
i=0 


lo. @) 


&* pre ® Hy) dx = | e-* pai) OTe) ax 


31.9 Evaluating Permanents 


Facts: 


For facts for which no specific reference is given and for background reading on the material in this 
subsection, see [Min78]. 


1. 


2: 


Since the permanent is not invariant under elementary row operations, it cannot be calculated by 
Gaussian elimination. 
(Ryser’s formula) If A = [a,j] is any n x n matrix, 


per(A) = S°(-1)" SO Po ay. 
r=1 


aEQ,n i=1 fea 
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3. [NW78] A straightforward implementation of Ryser’s formula has time complexity @(*2"). By 
enumerating the a in Gray Code order (i.e., by choosing an ordering in which any two consecutive 
a’s differ by a single entry), Nijenhuis/Wilf improved this to @(n2”). They cut the execution time 
by a further factor of two by exploiting the relationship between the term corresponding to a and 
that corresponding to {1,2,...,}\a. This second savings is not always desirable when calculating 
permanents of integer matrices, since it introduces fractions. 


Ryser/Nijenhuis/Wilf (RNW) Algorithm for calculating per(A) for A = [aj] of order n. 


p=-L 
fori from 1 ton do 
Xj = Gin — ae, Gijs 
P= Pp ¥X5 
gi = 95 
si=—]; 
for k from 2 to 2"~! do 
if k is even then 


j=); 
else 
j= 2; 
while g;_; = 0 do 
jr=jth 
Zi= 1-2 gj; 
gj = 1— gi 
Si= Ss; 
t:= $3 


fori from 1 to n do 
Xj = Xi + Z * Ais 
t:= t* Xj3 
pi=prh 
return(2(—1)"p) 


4. For sufficiently sparse matrices, a simple enumeration of nonzero diagonals by backtracking, or a 
recursive Laplace expansion, will be faster than RNW. 

5. A hybrid approach is to use Laplace expansion to expand any rows or columns that have very few 
nonzero entries, then employ RNW. 

6. [DL92] The calculation of the permanent is a #P-complete problem. This is still true if attention 
is restricted to matrices in A%. So, it is extremely unlikely that a polynomial time algorithm for 
calculating permanents exists. 

7. [Lub90] As a result of the above, much work has been done on approximation algorithms for 
permanents. 


31.10 Connections between Determinants and Permanents 


Definitions: 
For any partition A of n let x, denote the irreducible character of the symmetric group S,, associated 
with 4 by the standard bijection (see Section 68.6 or [Mac95, p. 114]) between partitions and irreducible 
characters. 

Let €, be the identity in S,. 
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The matrix function f, defined by 


1 n 
3(M) = —— XG) | | Mic), 
s xa (En) Xu I] m 
for each M = [mj] € C"*”, is called a normalized immanant (without the factor of 1/x,(€,) it is an 


immanant). 
The partial order < is defined on the set of partitions of an integer n by stating that 2 < jz means that 
f,(H) < f,(H) for all H € PD,. 


Facts: 


For facts for which no specific reference is given and for background reading on the material in this 
subsection, see [Mer97]. 


If = (n), then x,, is the principal/trivial character and f, is the permanent. 

If = (1"), then x, is the alternating character and f, is the determinant. 

[Sch18] fH) is a nonnegative real number for all H € PD, and all 2. 

[MM61] For n > 3 there is no linear transformation T such that per(A) = det (T(A)) for every 
A € R”*". In particular, there is no way of affixing minus signs to some entries that will convert 
the permanent into a determinant. 

5. [Lev73] For all sufficiently large n there exists a fully indecomposable matrix A € A? such that 
per(A) = det(A). 

6. [Sch18], [Mar64] If A = [aj] € PD,, then det(A) < []j_, aii < per(A). Equality holds iff A is 
diagonal or A has a zero row/column. 

7. [Sch18] For arbitrary A,if A € PD,, then f,(A) > det(A). In other words (1”) <A for all partitions 
d of n. 

8. [Hey88] The hook immanants are linearly ordered between det and per. That is, (1”) < (2, 1?) 
(3,177) a --» d(n—1,1) <(n). 

9. A special case of the permanental dominance conjecture asserts that A < (m) for all partitions 4 
of n. This has been proven only for special cases, which include (a) n < 13, (b) partitions with no 
more than three parts which exceed 2, and (c) the hook immanants mentioned above. 

10. For two partitions A, jz of n to satisfy A <1 2, it is necessary but not sufficient that jz majorizes i. 


aan ee 


Examples: 


1. Although jz = (3, 1) majorizes A = (2,2), neither A < yz nor 4 <A. This is demonstrated by taking 
A= J, @ Jz and B = J, ® J; and noting that f,(A) = 2 > ; = f, (A) but A(B) =0<2= 
Su (B). 
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An m xX n matrix W whose entries are all 1 or —1 is called a (+1)-design matrix; if the entries of W are 
0 or 1, then W is a (0, 1)-design matrix. Each design matrix corresponds to a weighing design. That is, 
a scheme for estimating the weights of n objects in m weighings. Since the weights of n objects cannot 
be estimated in fewer than n weighings, we consider only those pairs (m,n) with m > n. The rows of W 
encode a two-pan or one-pan weighing design with n objects x), ...,x, being weighed in m weighings. If 
W ¢ {+1}”*", an entry of 1 in the (i, j)-th position of W indicates that object x; is put in the right pan 
in the i-th weighing while an entry of —1 means that x; is placed in the left pan. If W € {0,1}"*", an 
entry of 1 in the (i, j)-th position indicates that object x; is included in the i-th weighing while an entry 
of 0 means that the object is not included. In the presence of errors for the scale, we can expect only to 
find estimators w,...,w,, for the actual weights w),...,w, of the objects. We want to choose a weighing 
design that is optimal with respect to some condition, an idea going back to Hotelling [Hot44] and Mood 
[Moo46]. See also [HS79] and [Slo79]. Under certain assumptions on the error of the scale, we can express 
optimality conditions in terms of W? W (see [Puk93]). The value of det W! W is inversely proportional 
to the volume of the confidence region of the estimators of the weights of the objects. Thus, matrices for 
which det W? W is large correspond to weighing designs that are desirable. 


32.1 Introduction 


This section includes basic definitions; facts and examples can be found in the following sections. 


Definitions: 


A matrix W € {+1} is a (+1)-design matrix; W € {0, 1}'"*" is a (0, 1)-design matrix. 
A matrix W € {+1}'*" (respectively, W € {0, 1}'"*") is called D-optimal if det W'W is maximal over 
all matrices in {41}”*” (respectively, {0, 1}’”*”). 
a(m,n) = max{det W'W|W e {41}"*"}. 
B(m,n) = max{det W'W|W e {0, 1}”*"}. 
We write a(n) and B(n) for a(n, ) and B(n, n). 
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32.2 The (+1) and (0,1) Square Case 


Definitions: 
For a matrix V € {0, 1}-)*“-), define 


e€ {+1}"*". 


For a matrix W € {+1}"*", define Vw € {0, 1} "~)*@-) by 


1 
Vw = 3 (WO) + Ini) 


where W(1) is obtained from W by deleting the first row and column. 

A signature matrix is a +1 diagonal matrix. 

A Hadamard matrix of order n is a matrix H, € {+1}"*" with H, HT = nI,. 

A 2-design with parameters (v, k, A) (also called a (v, k, 4)-design) is a collection of k-subsets B;, called 
blocks, of a finite set X with cardinality v, such that each 2-subset of X is contained in exactly blocks. 

The (+1)-incidence matrix W = (wj;;) ofa 2-design is a matrix whose rows are indexed by the elements 
x; of X and whose columns are indexed by the blocks B;. The entry w;; = —lifx; € Bj; and wjj = +1 
otherwise. 


Facts: 


1. For any W € {+1}"*", there exist signature matrices S), S) such that for W’ = (S, WS)), Wij = 
for j =1,...,n and W/, = —1 fori = 2,...,n. Wis D-optimal if and only if W’ is D-optimal. 

2. det Wy = 2”—! det V. 

3. [Wil46] The (+1) square case in dimension n is related to the (0, 1) square case in dimension n — 1 
by the previous two facts, and a(n) = 4"~'B(n — 1). Facts are stated here for a(n) only, since the 
facts for B(n) can be easily derived from these. 

4. [Had93], [CRC96], [BJL93], [WPS72] Hadamard matrices 


* A necessary condition for the existence of a Hadamard matrix of order n is n = 1,n = 2, or 
n= 0 (mod 4). 


* Let H,, and H,, be two Hadamard matrices of orders m and n, respectively. Then H,, ® Hy, is a 
Hadamard matrix of order mn. 


* There exist infinitely many values n for which a Hadamard matrix H,, exists. 
* It is conjectured that for all n = 4k there exists a Hadamard matrix H,. 


* The smallest n for which the existence of a Hadamard matrix is in question (at the time of the 
writing of this chapter) isn = 668. 


5. [Had93], [CRC96], [BJL93], [WPS72] D-optimal (+1)-matrices: the case n = 4k 
* (4k) < (4k). 


* Anecessary and sufficient condition for equality to occur in this case is the existence ofa Hadamard 
matrix of order n. 


6. [Bar33], [Woj64], [Ehl64a], [Coh67], [Neu97] D-optimal (+1)-matrices: the case n = 4k + 1 
° w(4k + 1) < (8k + 1) (4k)*. 


* For equality to occur in this case, it is necessary and sufficient that 8k + 1 is the square of an 
integer and that there exists a matrix W € {+1}”*" with W7W = (n—1)In + Jn- 
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* Equality occurs for infinitely many values of n = 4k + 1. A.E. Brouwer ([Bro83]) constructed 
an infinite family of 2-designs with parameters (n = 2q? + 2q + 1,q7, (q? — q)/2). The (+1)- 
incidence matrix W,, of such a design satisfies wr W, = (n-VWInt+ Tn. 

* The results in [MK82] and [CMK87] provide upper bounds, which are stronger than (8k + 
1)(4k)** in case 8k + 1 is not the square of an integer. 

7. [Ehl64a], [Woj64] D-optimal (+1)-matrices: the case n = 4k + 2 

° a(4k +2) < (8k +2)? (4k)*. 

* For equality to occur in this case, it is necessary that n — 1 is the sum of two squares and that 
there exists a matrix W, € {+1}"*" such that 


a [= 2) +2 0, 5 
ww, = 32.1 
On (1 —2)In +2] 


* It is conjectured that the bound is attained whenever this is the case. 

* The bound is attained infinitely often. 

* Ifn — 1 isa square and there exists a matrix Ws € {+1}2*2 such that Wr We = a In + Ja, 
then construct the matrix W,, = W: @ Hp where 


A i . Then W, € {£1}"*” satisfies Equation (32.1) and attains the bound. Such a 


H= 
Lf 


matrix Ws exists if $ = 2q* + 2q +1. 
8. [Ehl64b] D-optimal (+1)-matrices: The case n = 4k + 3 


a(4k +3) < (4k)**+3-5 (4k + 4r)"(4k +44 4r)” (1 ur v(r +1) ; 


4k + 4r 4k+4+4r 
where s = 5 fork = 1,s = 5o0r6 fork =2,s = 6 for3 < k < 14,ands =7 fork > 15 and 
where r = |(4k + 3)/s|,4k +3 =rs+v,andu=s-—v. 
¢ This case is the least well understood of the four. 
* For equality to occur for n > 63, it is necessary that n = 1127? + 28j +7 and that there exists a 
matrix W,, € {£1}"*" with 
Wy Wy = Tp @ |(n— 3)Ie +4] 2] — Tn. 


* However, it is not known if this bound is attainable for any n > 63. 


* The best lower bound seems to be the one in [NR97]. In [NR97], an infinite family of matrices 
is constructed whose determinants attain about 37% of the bound above. 


9. See [OS] for (+1)-matrices with largest known determinant for n < 103. 


Examples: 


1. The following matrices are Hadamard matrices in {+1}*** and {+1}?*?: 


1 1 1 1 

1 -l 1 -l 
AM= 

1 1 —-l -l 

ToL) = 1 
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1 1 b=) =k =] 1 1 Leh =] 
1 1 -l 1 -l 1 -l 1. 1 1 -l 
1 1 —-1l -l 1 1 1 —-1 -1 -l 1 
—-1 -l | od | 1 1 1 1 1 1 
—l ol =I] Lest ol de Sd. eh eT eh 
—-1 -l 1 -1l -l 1 —-1l -l Lo 1. 1 
—l 1 1 1 1 1 1 1 1 1 1 1 
1 -l 1 -l Toeel coeds Sb 1 1 st 1 
1 PS Ste ed 1 —-l -1 -l 1 1 -l 
1 1 1 1 1 1 1 1 1 -l 1 1 
—1 1 —-1 -1 -1 -l 1 -l 1 1 -l 1 
-1 -l bok SS 1 1 —-l 1 } =! 


HHI = 4l, and HyHs = 12I. 
2. Let A= Js — 2I5. Then A’ A = 4]; + Js and det(A’ A) achieves the upper bound in Fact 6. 
3. Let 


A 
W= = AQ@ Hye {+1} 1*10) 
Le aos h € {t]} 
where A = J; — 215. Then 
8Is + 2Js 0 
WIW= i 
0 8I5+2Js 


and, hence, det( WW) achieves the upper bound in Fact 7. 

4, To obtain the upper bound in Fact 6 for n = 13, let V € {0,1}'*} be the (0,1) line-point 
incidence matrix for a projective plane of order 3. Then V'V = 31,3 + Ji3 and the matrix W = 
Ji3 —2V € {41} ®*® satisfies W7W = 1213 + Jy3 and its determinant attains the upper bound. 

5. Forn > 1landn = 3 (mod 4), no (+1)-matrix is known to have a determinant that equals the up- 
per bound in Fact 8. However, the following matrix W € {+1}'°*!°, which is listed in [OS], satisfies 


det(W!W) = 174755568785817600, 
which is about 94% of the upper bound 185454889323724800 in Fact 8 with k = 3,5 = 6. 


-1l -1 -1 -l 1 1 1 i 1 -l 1 1 1 1 -l 
=) sh 1 -l 1 1 -l 1 1 1 1 -l 1 1 
= Sle Sl 1 I. =] 1 1 —-l 1 1 -l 1 1 1 
—1 1 1 —-l -l 1 1 1 —-l 1 1 —-1l -1 -1 -l 
1 1 -l 1 —-1 -l 1 1 1 -l le Sh el, Hh 
1 -l 1 -l 1 -l 1 -l 1 1 1 —-1l -1 -1 -l 

1 1 1 1 1 1 =1 1 1 1 1 —-1l -l 1 i | 
W=,-1 1 1 1 1 1 1 1 1 -l 1 -l 1 1 1 
1 1 1 —-1l -1 -l 1 1 1 1 -l 1 1 1 1 

1 -l 1 —-l -1 -1 -l Lol 1 1 1 -l 1 1 

1 1 1 1 1 1 1 1 1 1 1 1 1 ja Sh 

1 1 —-1l -l ak 1 1 1 1 1 1 1 1 1 1 
1 -l 1 1 -l 1 1 1 1 1 1 1 1 1 -l 
—l1 1 1 1 1 -l boy Shoal Sd 1 -l 1 -l 
1 1 1 1 1 1 bh. =) 1 1 1 1 -l 1 
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32.3. The (+1) Nonsquare Case 


Facts: 
1. [Had93] a(4k, n) < (4k)". 
2. [Pay74] a(4k + 1,n) < (4k +n)(4k)"1. 
3. [Pay74] 


(4k + n)?(4k)"~?, if n is even, 
a(4k + 2,n) < (32.2) 
(4k +n+ 1)(4k +n —1)(4k)", if nis odd. 


4. [GK80b] If m = 4k — 1 > 2n —5, thena(m,n) < (4k — n)(4k)""1. 


Examples: 

1. Ifm = 4k, equality can be achieved by taking Wo to be the matrix consisting of any n columns of a 
Hadamard matrix of order 4k. Then wo Wo = 4kI,, and, hence, det we Wo = a(4k, n). 

2. Ifm = 4k + 1, adjoin a row ofall 1s to Wy and call the new matrix W,. We have W, € {41}44+)xn 
and W/W, = ml, + J. Hence, det W) W, = a(4k + 1,1). 

3. Ifm= 4k +2, letr) = (1,1,...,1) andr2 = (1,... ,1,—1,... ,—1) be n-tuples with r, -r. = 0, 
ifn is even, andr; -r2 = 1, ifn is odd. Adjoin rows r; and rz to Wo. Call the resulting matrix W). 
Then Wy € {41} 442)*" and 


4k I, + 2); 0; 
Wy Wy = ifn = 21 is even 
0; 4k I; + 2], 
and 
4kIi4) + 2Si41 Ora 
ww | ss ' ifn = 21 + Lis odd. 
O1i+1 4kI, + 2J) 


Thus, det wi Ww, = a(4k + 2,n). 

4. If m = 4k — 1, we may assume without loss of generality that the first row of Wo is an all 1s row. 
Remove this first row of Wo and call that matrix W_,. Note that W_; € {£1}“*-)*" and that 
W!,W_, = 4kI, — Jn. Hence, det W7, W_; = a(4k — 1,n). 

5. It is not necessary to have a Hadamard matrix H,, of order m. All we require is the existence of a 
matrix W € {+1}***" with W'W = ml,,. See [GK80a] for details. 

6. Upper bounds on a(4k — 1,1) when m = 4k — 1 < 2n — 5 are given in [GK80b], [KF84], [SS91]. 


32.4 The (0,1) Nonsquare Case: Regular 
D-Optimal Matrices 


Definitions: 
Let W be a (0, 1)-design matrix in {0, 1}'"*”". For n odd, W is balanced if every row of W has exactly 


(n+ 1)/2 ones; for n even, W is balanced if every row of W has exactly n/2 ones or exactly (n + 2)/2 ones. 
A design matrix W e€ {0, 1}"*" is regular if it is balanced and W7W = ¢(J + J) for some integer t. 
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1. mea If n is odd, then B(m,n) < (n+ 1) ) (“4 ntl)m ny", with equality if and only if W is regular. 


2. [NWZ97] If 1 is even, then B(m,n) < (n+ 1) ( ner Vi, with equality if and only if W is regular. 


3. [NWZ98a] A regular matrix exists in {0, 1}’"*" only if 


2(n + 1) dividesmforn=0O (mod 4), 
2n divides mforn =1 (mod 4), 

n+ 1 divides mforn=2 (mod 4), 

n divides m forn =3 (mod 4). 


4. [NWZ98a] Ifn = 4t—1and H € {0, 1}”"*" is the incidence matrix for a (4t— 1, 2t—1, t—1)-design, 
k 


then W = J — H isaregular D-optimal matrix and [W!,---,W' ]Tisa regular D-optimal matrix 
k 


Cae ae 
in {0, 1}*"*", Let W, be the matrix obtained by deleting any column from W. Then [W/,--- , W. ]7 
is a regular D-optimal matrix in {0, 1}#"*"-)), 
5. [NWZ98a] If n = 4t + 1 is a power of a prime integer, then a D-optimal regular matrix W, € 


F exists. Let W3 € {0, —) be the matrix obtaine eleting any column from W5. 
{0, 1}2"" exists. Let Ws € (0, 1}2**— be the matrix obtained by deleting any column from W; 
k k 
—__. —-_—_ 
Then [Wy,--- , Wy]? and [WJ,--- , Wj]? are regular D-optimal matrices. 


Examples: 


1. Let n = 4and m = 10. The following matrix is balanced and regular: 


1 11 041d1éi100 (0 6 3 3 3 
- 1103110 0ii1i2i1 «+0 7 3 6 3 3 
w= . ww= : 
101 i10%t10ii10i«1 3 3 6 3 
011310021 0i21i41 3. 3 3 6 


The inequality in Fact 2 is attained at W: 
6-10 
B(10,4) =5 ( 7s ) = 405 = det(W7W). 


Thus W is D-optimal. 

2. A regular matrix exists for the case n = 9,m = 18. (Fact 3, where n = 1 (mod 4) and 2n = 18 
divides m = 18.) The regular matrix is constructed from the Galois field GF(9) with nine elements. 
Choosing GF(9) = Z3/ (x* + 1), the element 6 = x +2 of order 8, generates the nonzero elements 
in GF(9). The nonzero quadratic residues of GF(9) are Q = {1, 67, 6*, 0°}. Define K, € {0, 1}°*? 
by 


1, ifrep+Q, 
(Ki)p,r = . . 
0, if otherwise, 
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where the rows and columns p,t are indexed by {0, 1,0, 67,...,07}. Then 
01 0101 0 1 ~0 
10000141 0 «41 
000 1 1 1 0 0 1 
10310 00 01 41 
K,;=/0 010011410 
11 10410 0 0 0 
01 0010011 
10011 0 1 0 0 
01 1 21 00 1 0 0 


Define K, in the same way but with the nonzero quadratic nonresidues R = {0, 67,6°.07} in place 
of Q. Then the matrix [K,, Ky] € {0, 1}°%!® satisfies 


K\ Kj + KK} =515 + 3Jo. 


Let W = [Jo — K, Jo — K2]. Then W is a D-optimal regular design matrix: WW! = 5(Iy + Jo). 
3. Let t = 2 and n = 7. The following matrix H is the incidence matrix for a (7, 3, 1)-design: 


II 
or OF OF OS 


1 0 
0 0 
0 1 
1 1 
1 0 
0 0 
0 1 


Fe OF CO OF CO 
ere Oo Oo co oO eS 
or fF CO KF CO CO 


1 
1 
1 
0 
0 
0 
0 


Then (Fact 4) W = J; — H isa regular D-optimal matrix in {0,1}’*? and [W’,..., W7]" isa 
regular D-optimal matrix in {0, 1}7**’. 


32.5 The (0,1) Nonsquare Case: Nonregular 
D-Optimal Matrices 


It is clear from Fact 3 in section 32.4 that for most pairs (m,n), no regular D-optimal matrix exists. For 
example, if n = 7, then the only values of m for which a regular D-optimal matrix exists are m = 7t. Thus, 
for m = 7t +1, with 0 < r < 6, a D-optimal matrix cannot be regular unless r = 0. The only values of n 
for which 6(m, n) is known for all values of m are n = 2, 3, 4,5, 6. 


Facts: 
1. [HKL96] n = 2, m = 3t +r withr = 0,1,2: 


ne) 3t? , forr=0 
m,2) = 
3r+2t, forr=1 


2. [HKL96] n = 3,m = 3t +r withr = 0, 1,2: 


B(m,3) = 40" (t+ DY. 
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3. [NWZ98b] n = 4,m = 10t+r withO <r < 9: 


405 t* , forr=0 
810 (24+5t) , forr=1 
3t(1+3t)* (24151) , forr=2 
34(1+3t) (8+ 150) , forr=3 
3(14+3t)> (+52) , forr=4 
B(m, 4) = 5 ; 
(1+3t)° (19+60t+45t?) , forr=5 
3(24+3¢)? (2452) , forr=6 


3 (1 +t) (24+3t) (74+15t) , forr=7 
3 (1 +t) (24+3t) (134+15t), forr=8 
81(1+1t) (3452) , forr=9 


4. [NWZ98b] In the case n = 5, all D-optimal matrices are balanced except when m = 5,6, 7,8, 15, 
16, 17,27. For m = 10t +r with 0 < r < 9 and m not equal to any of the exceptional values, we 


have 
1458 f° , forr=0 
729 t* (1+2t) , forr=l 
1620 (1+3f) (2+3t) , forr=2 
27? (14+3t) (5+6t) , forr=3 
54t (1+) (1+3t) , forr=4 
as 9(1+t) (1430) (14+15t+182), forr=5 
54(14+ 1)? (1431) , forr=6 
27(14+t) (1+3t) (5+6t) , forr=7 
162(1 +t)? (1+3t) (2+3f) , forr=8 
729(1+t)* (1+2t) , forr=8 


The values of 8(m, 5) for the eight exceptional values of m are 


B(5,5) = 25 B(6,5) = 64 B(7,5) = 192 B(8,5) = 384 
B(15,5) = 9880 (16,5) = 13975 (17,5) = 19500 (27,5) = 202752. 


Each of these is greater than the value of the corresponding polynomial above. For example, if 
m = 16 so that t = 1 andr = 6, then 54(1 +t)? (1 +31)? = 13824, which is less than 13975. 

5. [NWZ00] For n = 6, all D-optimal matrices are balanced except when m = 6, 8,9, 13. For m = 
7t +r withO <r <6andm+6,8,9, 13: 


448 t° , forr=0 
16¢t* (1+2t) (54+ 14t) , forr=l 
4° (14+2t) (34142) , forr=2 
B(m, 6) = (1424) (14+ 142) , forr=3 
(1+2t)° (134 141) , forr=4 


4(1+t)? (1421) (114140), forr=5 
16(1+1)* (14+ 2t) (94+ 14t), forr=6 
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The values of 8(6, m) for the four exceptional values of m are 
B(6,6)=81  A(8,6)=832 A(9,6)=1620 (13,6) = 16512. 


As in the case for n = 5, the values of 6(m, 6) exceed the value of the corresponding polynomial. 


Examples: 


1. Design matrices are exhibited for the above cases in the sources listed. For example, if n = 4, let 


is he i 8a 
1104110042121 «+0 

W = > 
101 1 0 41 021 01 
01 21 21 003104121 

t 
a 
v; be the ith row of Wo. Ifr = 3, then the matrix [WJ,--- , Wd ,v2,v2,v4]" isa D-optimal matrix 


in {0, 1443) x4 


32.6 The (0,1) Nonsquare Case: Large m 


Facts: 


1. [NWZ98a] For each value of n, all D-optimal matrices in {0,1}’”*" are balanced for sufficiently 
large values of m. 

2. [NW02], [AFN03] In addition to the values n = 2,3,4,5,6, for which 6(m,n) is known for all 
m, the only other values of n for which 6(m, 1) is known for all sufficiently large values of m are 
n = 7,11, 15, 19, 23, 27. 


Examples: 
1. [NW02] 
B(7t+7r,7) = 207 T(t +, 
for sufficiently large values of m = 7t +r. 


32.7 The (0,1) Nonsquare Case: n = —1 (mod 4) 


The theory for D-optimal (0, 1)-designs is most developed for the cases where n = —1 (mod 4). 


Definitions: 


For an n x n matrix A, the trace-sequence A is (trace(A), trace(A”),--+ , trace(A")). 

G(v, 65) is the set of all 6-regular graphs on v vertices. 

Let graph G be a graph in G(v,65) and let Ag be the adjacency matrix of G. The graph G is trace- 
minimal if the trace-sequence of its adjacency matrix (trace(Ag), trace(.A2,), +--+ , trace(AZ)) is least in 
lexicographic order among all graphs in G(v, 5). 


Facts: 


1. [AFNO03] If n = —1 (mod 4), then for each 0 < r < n and all sufficiently large values of f, 
B(nt + 1,n) is a polynomial in t of degree n. These polynomials are related to the adjacency 
matrices Ag of certain regular graphs G. 
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2. [AFN03], [AFN06] The polynomial 6(nt + r,n) depends on a trace-minimal graph in G(v, 4). 
Once a trace-minimal graph G is found in the appropriate graph class G(v, 4), the polynomial 
B(nt + 1r,n) can be computed. There are four theorems [AFN03] governing this situation; one for 
each congruence class of r (mod 4). 

3. [AFNO3] Trace-minimal graphs are known for all of the graph classes necessary to obtain formulas 
for B(nt + 1,n) for n = 3,7, 11,15, 19,23, and 27 and ¢ sufficiently large. 

4. [AFN06] Let G bea connected strongly regular graph with no three cycles. Then G is trace-minimal. 

5. [AFN06] The following graphs are trace-minimal in their graph class: 


Graph Class G 

G(v,0) Graph with v vertices and no edges 

G(2v, 1) vK>, a matching of 2v vertices 

G(v, 2) C,, the cycle graph on v vertices 

G(2v,v) K,,,, the complete bipartite graph with v vertices in each set of the bipartition 


G(2v,2v—2) Kz, —vK>2, the complement of a matching 


G(v,v —1) K,, the complete graph on v vertices 


6. [AFN06] Let G bea connected regular graph with girth g such that Ag hask+ 1 distinct eigenvalues. 
If g is even, then g < 2k with equality only if G is trace-minimal. If g is odd, then g < 2k + 1 with 
equality only if G is trace-minimal. 


Examples: 


1. Letn = 4p —1=~—1 (mod 4) andr = 4d +2 =2 (mod 4). Let G be a trace-minimal graph in 
G(2p, p +d). Then 


4t|pa,(pt+d)} 


B(nt +r,n) = (1p : 


for sufficiently large values of t. Taking n = 15,r = 10, we have p = 4,d = 2. The appropriate 
graph class is G(8,6). There is only one graph G in this class, namely the complement of the 
matching 4K}. Thus, it is trace-minimal. Since p.4,(x) = (x — 6)x*(x + 2)%, 


4t( pa, (4t + 2)]}? 


B(15t + 10, 15) = ate 


= 16(4t)(4t + 2)°(4t + 4)°, 
for sufficiently large tf. 

2. Letn = 4p —1=-—1 (mod 4) andr = 4d +1 =1 (mod 4). Let G bea trace-minimal graph in 
G(2p,d). Then 


A(t + 1)[pag (pt + d)]* 
t2 


Bint +r,n) = 


> 


for sufficiently large t. Taking n = 15,r = 9 we have p = 4,d = 2. The appropriate graph class is 
G(8, 2). There are three (nonisomorphic) graphs in this class: Cg, Cs; U C3, and C4 U C4, where Cx 
stands for a k-cycle graph. The trace sequences for these three graphs are 


(0, 16, 0, 48, 0, 160, 0, 576) , forG=Ceg 
(trace(Ag), trace(Az), see , trace(A%)) = ¢ (0, 16, 6, 48, 40, 166, 196,608), for G =C;U C3 
(0, 16, 0, 64,0, 256,0,1024) , forG =CyUCy. 


D-Optimal Matrices 32-11 


FIGURE 32.1 


Thus, Cg is the only trace-minimal graph in the graph class G(8, 2). The characteristic polynomial 
for Ac, is 


(x — 2)x?(x + 2)(x? — 2). 
Thus, 


A(t + 1)[pag(4t + 2)]? 
t2 


B(i5t + 9,15) = = 16(4t + 2)*(4t + 4)°(16t7 + 16t + 2)*. 


3. The Petersen graph (Figure 32.1) is an example of a strongly regular graph. (See Fact 4.) It is 
trace-minimal in G(10, 3): 

4. Let P bethe projective geometry with seven points, 1, 2, 3, 4, 5, 6, 7 and seven lines, 123, 147, 156, 257, 
246, 367, 345 (Figure 32.2): The line-point incidence matrix for P is: 


Zz 

II 
on 
Co OFF oO Oe 
ere Ooo co oO KF 
- Of OF oO KS O&O 
(a = Ee ee oa oo ee 
or FF OF CO Ff 
or OO fF CO KS OS 


4 


FIGURE 32.2 
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Let G be the incidence graph of P having 14 vertices and adjacency matrix given by 
0 N 
Ag = ore 
Then G is trace-minimal by Fact 6: G is a regular graph of degree 3. The girth of G is g = 6. The 


characteristic polynomial of Ag is (x — 3)(x + 3)(x? — 2)® and so Ag hask + 1 = 4 distinct 
eigenvalues. Since 2k = g, it follows that G is trace-minimal in G(14, 3). 


32.8 Balanced (0, 1)-Matrices and (-1)-Matrices 


Let n = 2k — 1 be odd. There is a connection between balanced (0, 1)-design matrices and (+1)-design 
matrices. 


Facts: 


1. [NWZ98a] Let W be a balanced design matrix in {0, 1}’”*", so that each row of W contains exactly 
k ones and k — 1 zeros. Let q be a positive integer and 


n Orewa ad 
Low =| He hk 


Ja J qam 


Then det L(W)? L(W) = q4” det W'W. It follows that for sufficiently large m, if W is a balanced 
(0, 1)-design matrix and L(W) is a D-optimal design matrix, then W is also D-optimal. 
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The origins of sign pattern matrices are in the book [Sam47] by the Nobel Economics Prize winner 
P. Samuelson, who pointed to the need to solve certain problems in economics and other areas based 
only on the signs of the entries of the matrices. The study of sign pattern matrices has become somewhat 
synonymous with qualitative matrix analysis. The dissertation of C. Eschenbach [Esc87], directed by 
C.R. Johnson, studied sign pattern matrices that “require” or “allow” certain properties and summarized 
the work on sign patterns up to that point. In 1995, Richard Brualdi and Bryan Shader produced a thorough 
treatment [BS95] on sign pattern matrices from the sign-solvability vantage point. There is such a wealth 
of information contained in [BS95] that it is not possible to represent all of it here. Since 1995 there 
has been a considerable number of papers on sign patterns and some generalized notions such as ray 
patterns. We remark that in this chapter we mostly use {+, —, 0} notation for sign patterns, whereas in the 
literature {1, —1,0} notation is also commonly used, such as in [BS95]. We further note that because of 
the interplay between sign pattern matrices and graph theory, the study of sign patterns is regarded as a 
part of combinatorial matrix theory. 


33.1 Basic Concepts 


Definitions: 
A sign pattern matrix (or sign pattern) is a matrix whose entries come from the set {+, —, 0}. For a real 
matrix B, sgn(B) is the sign pattern whose entries are the signs of the corresponding entries in B. 

If A is an m x n sign pattern matrix, the sign pattern class (or qualitative class) of A, denoted Q(A), 
is the set of all m x n real matrices B with sgn(B) = A. If C is a real matrix, its qualitative class is given 


by Q(C) = Q(sgn(C)). 
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A generalized sign pattern A is a matrix whose entries are from the set {+, —, 0, #}, where # indicates 
an ambiguous sum (the result of adding + with —). The qualitative class of A is defined by allowing the 
# entries to be completely free. Two generalized sign patterns are compatible if there is a common real 
matrix in their qualitative classes. 

A subpattern A of a sign pattern A is a sign pattern obtained by replacing some (possibly none) of the 
nonzero entries in A with 0; this fact is denoted by AXA. 

A diagonal pattern is a square sign pattern all of whose off-diagonal entries are zero. Similarly, standard 
matrix terms such as “tridiagonal” and “upper triangular” can be applied to sign patterns having the 
required pattern of zero entries. 

A permutation pattern is a square sign pattern matrix with entries 0 and +, where the entry + occurs 
precisely once in each row and in each column. 

A permutational similarity of the (square) sign pattern A is a product of the form P? AP, where P is 
a permutation pattern. 

A permutational equivalence of the sign pattern A is a product of the form P| AP), where P, and P, 
are permutation patterns. 

The identity pattern of order n, denoted I, is the n x n diagonal pattern with + diagonal entries. 

A signature pattern is a diagonal sign pattern matrix, each of whose diagonal entries is + or —. 

A signature similarity of the (square) sign pattern A is a product of the form S AS, where S is a signature 
pattern. 

If P is a property referring to a real matrix, then a sign pattern A requires P if every real matrix in Q(A) 
has property P, or allows P if some real matrix in Q(A) has property P. 

The digraph of ann x n sign pattern A = [a;;], denoted I'(A), is the digraph with vertex set {1, 2,..., n}, 
where (i, j) is an arc iff aj; 0. (See Chapter 29 for more information on digraphs.) 

The signed digraph of an n x n sign pattern A = [a;;], denoted D(A), is the digraph with vertex set 
{1,2,...,}, where (i, j) is an arc (bearing aj; as its sign) iffaj; A 0. 

If A = [a;;] isan n x n sign pattern matrix, then a (simple) cycle of length k (or a k-cycle) in A is 
a formal product of the form y = 4j,j,j,i, . . . di,i,, where each of the elements is nonzero and the index 
set {1),i2,... ,1%} consists of distinct indices. The sign (positive or negative) of a simple cycle in a sign 
pattern A is the actual product of the entries in the cycle, following the obvious rules that multiplication 
is commutative and associative, and (+)(+) = +, (+)(—) = -. 

A composite cycle y in A is a product of simple cycles, say y = 7172... Ym where the index sets of 
the y;’s are mutually disjoint. If the length of y; is /;, then the length of y is 5>;”_, 1;, and the signature 
of y is (—)UaG-V, A cycle y is odd (even) when the length of the simple or composite cycle y is odd 
(even). 

If A = [a;j] isan n x n sign pattern matrix, then a path of length k in A is a formal product of the form 
j,i, Aini, -» - Gizi,,,» Where each of the elements is nonzero and the indices i),i2,... ,ix41 are distinct. 


Facts: 


1. Simple cycles and paths in an n x n sign pattern matrix A correspond to simple cycles and paths 
in the digraph of A. In particular, the path 4;,;,4j,;, ...di,i,,, in A corresponds to the path i, > 
iy > ...—> igs) in the digraph of A. 

2. If Aisann x n sign pattern, then each nonzero term in det(A) is the product of the signature of a 
composite cycle y of length n in A with the actual product of the entries in y. 

3. Two generalized sign patterns are compatible if and only if the signs of each position whose sign is 
specified in both are equal. 


Examples: 6 sb ed oe 


1. The matrix is in Q(A), where A = . 
ee ee, —- -— + 
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fo SE Or, Se 
On er Se ee 
2. If A= [aij] = |+ : 
--+- 4 
a 
then the composite cycle y = (a)242343))(a45a54) has length 5 and negative signature, and yields 
the term —4@12423431]d45a54 = — in det(A). 
3. IfA= a ot , then A? = ve , which is compatible with i; 7 . 
_ # + 0 # 


33.2 Sign Nonsingularity 


Definitions: 


A square sign pattern A is sign nonsingular (SNS) if every matrix B € Q(A) is nonsingular. 

A strong sign nonsingular sign pattern, abbreviated an S*NS-pattern, is an SNS-pattern A such that 
the matrix B~! is in the same sign pattern class for all B € Q(A). 

A self-inverse sign pattern is an S*NS-pattern A such that B~! € Q(A) for every matrix B € Q(A). 

A maximal sign nonsingular sign pattern matrix is an SNS-pattern A where no zero entry of A can be 
set nonzero so that the resulting pattern is SNS. 

A nearly sign nonsingular (NSNS) sign pattern matrix is a square pattern A having at least two nonzero 
terms in the expansion of its determinant, with precisely one nonzero term having opposite sign to the 
others. 

A square sign pattern A is sign singular if every matrix B € Q(A) is singular. 

The zero pattern of a sign pattern A, denoted | Al, is the (0,+)-pattern obtained by replacing each — 
entry in A bya+. 

Since a sign pattern may be represented by any real matrix in its qualitative class, many concepts defined 
on sign patterns (such as SNS and S*NS) may be applied to real matrices. 


Facts: 
Most of the following facts can be found in [BS95, Chaps. 1—4 and 6-8]. 


1. The n x n sign pattern A is sign nonsingular if and only if det(A) = + or det(A) = —, that is, 
in the standard expansion of det(A) into n! terms, there is at least one nonzero term, and all the 
nonzero terms have the same sign. 

2. Annxn pattern A isan SNS-pattern iff for any n x n signature pattern D and anyn x n permutation 
patterns P,, P,, DP, AP, is an SNS-pattern. 

3. [BMQ68] For any SNS-pattern A, there exist a signature pattern D and a permutation pattern P 
such that DP A has negative diagonal entries. 

4. [BMQ68] An n x n sign pattern A with negative main diagonal is SNS iff the actual product of 
entries of every simple cycle in A is negative. 

5. [Gib71] Ifan n x n, n > 3, sign pattern A is SNS, then A has at least Cs) zero entries, with 
exactly this number iff there exist permutation patterns P,; and P such that P; AP» has the same 
zero/nonzero pattern as the Hessenberg pattern given in Example 1 below. 

6. The fully indecomposable maximal SNS-patterns of order < 9 are given in [LMV96]. [GOD96] An 
n X n sign pattern A is a fully indecomposable maximal SNS-pattern with C34 zero entries iff A is 
equivalent (namely, one can be be transformed into the other by any combination of transposition, 
multiplication by permutation patterns, and multiplication by signature patterns) to the pattern 
given in Example 1 below. Forn > 5, there is precisely one equivalence class of fully indecomposable 
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maximal SNS-patterns with ee + 1 zero entries, and there are precisely two such equivalence 


classes having (",') + 2 zero entries. 


7. [BS95, Corollary 1.2.8] If A isan n x n sign pattern, then A is an S*NS-pattern iff 


(a) Ais an SNS-pattern, and 


(b) For each i and j with a;; = 0, the submatrix A(i, j) of A of order n — 1 obtained by deleting 


row i and column j is either an SNS-pattern or a sign singular pattern. 


8. [BMQ68] If A is an n x n sign pattern with negative main diagonal, then A is an S?NS-pattern iff 


(a) The actual product of entries of every simple cycle in A is negative, and 


(b) The actual product of entries of every simple path in A is the same, for any paths with the same 


initial row index and the same terminal column index. 


9. [LLM95] An irreducible sign pattern A is NSNS iff there exists a permutation pattern P and a 


signature pattern S such that B = APS satisfies: 


(a) bj; < Ofori =1,2,...,n. 
(b) The actual product of entries of every cycle of length at least 2 of D(B) is positive. 


(c) D(B) is intercyclic (namely, any two cycles of lengths at least two have a common vertex). 


10. [Bru88] An n x n sign pattern A is SNS iff per(|B|) = |det(B)|, where B is the (1, —1, 0)-matrix 


in Q(A) and |B| is obtained from B by replacing every —1 entry with 1. 


11. [Tho86] The problem of determining whether a given sign pattern A is an SNS-pattern is equivalent 


to the problem of determining whether a certain digraph related to D(A) has an even cycle. 


For further reading, see [Kas63], [Bru88], [BS91], [BC92], [EHJ93], [BCS94a], [LMO96], [SS0O1], 


and [SS02]. 


Examples: 


1. For n > 2, the following Hessenberg pattern is SNS: 


- + 0 0 0 
- - + 0 
Bie. te 0 0 
H, = 
+ 


oo + 
o+ | 
+ | o 
°o 


+ oO 
oo 
—) 
o+ 
+ | 
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3. 
-— + 0 O 
0 — + 0 
A= 
0 0 -— + 
= 0 — 
is S7NS with inverse pattern 
+ a a = 
+ + - - 
+ ++ - 


- + 0 + 
— + 0 

—- —- + 0 
> Sy Rs 

0 - — + 

= Qi ee ee 


33.3 Sign-Solvability, L-Matrices, and S*-Matrices 


Definitions: 


A system of linear equations Ax = b (where A and b are both sign patterns or both real matrices) is 
sign-solvable if for each A € Q(A) and for each b € Q(b), the system Ax = b is consistent and 


{% : there exist A € Q(A) and b € Q(b) with Ax = b} 


is entirely contained in one qualitative class. 

A sign pattern or real matrix A is an L-matrix if for every B € Q(A), the rows of B are linearly 
independent. 

A barely L-matrix is an L -matrix that is not an L-matrix if any column is deleted. 

Ann x (n+ 1) matrix B is an S*-matrix provided that each of the n + 1 matrices obtained by deleting 
a column of B is an SNS matrix. 

Ann x (n+ 1) matrix B is an S-matrix if it is an S*-matrix and the kernel of every matrix in Q(B) 
contains a vector all of whose coordinates are positive. 

A signing of order k is a nonzero (0, 1, —1)- or (0, +, —)-diagonal matrix of order k. 

A signing D' = diag(d{, dj, ...,dj,) is an extension of the signing D if D' # D and d/ = d; whenever 
d; #0. 

A strict signing is a signing where all the diagonal entries are nonzero. 

Let D = diag(d), dp,...,d,) be a signing of order k and let A be an m x n (real or sign pattern) matrix. 
If k = m, then DA is a row signing of the matrix A, and if D is strict, then DA is a strict row signing 
of the matrix A. If k = n, then AD is a column signing of the matrix A, and if D is strict, then AD isa 
strict column signing of the matrix A. 

A real or sign pattern vector is balanced provided either it is a zero vector or it has both a positive entry 
and a negative entry. A vector v is unisigned if it is not balanced. A balanced row signing of the matrix 
A is a row signing of A in which all the columns are balanced. A balanced column signing of the matrix 
A is a column signing of A in which all the rows are balanced. 
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Most of the following facts can be found in [BS95, Chaps. 1-3]. 


1; 


10. 


11. 
12. 


13. 


14. 


15. 
16. 


A square sign pattern A is an L-matrix iff A is an SNS matrix. 


2. The linear system Ax = 0 is sign-solvable iff A? is an L -matrix 
3. 
4. Ax = b is sign-solvable for all b if A is a square matrix and there exists a permutation matrix P 


If Ax = b is sign-solvable, then A! is an L-matrix. 


such that PA is an invertible diagonal matrix. 

Sign-solvability has been studied using signed digraphs; see [Man82], [Han83], [BS95, Chap. 3], 
and [Sha00]. 

[BS95] Let A be a matrix of order n and let b be ann x 1 vector. Then Ax = b is sign-solvable iff 
A is an SNS-matrix and for eachi, 1 <i <n, the matrix A(i < b), obtained from A by replacing 
the i-th column by J, is either an SNS matrix or has an identically zero determinant. 

[BS95] If AX = B is a sign-solvable linear system where A and B are square matrices of order n 
and B does not have an identically zero determinant, then A is an S°NS matrix. 

[BS95] Let Ax = b bea linear system such that A has no zero rows. Then the linear system Ax = b 
is sign-solvable and the vectors in its qualitative solution class have no zero coordinates iff the 
matrix [A | — b] is an S*-matrix. 

Ann x (n+ 1) matrix B is an S*-matrix iff there exists a vector w with no zero coordinates such 
that the kernels of the matrices B € Q(B) are contained in {0} U Q(w) U Q(—w). 

[Man82] and [KLM84] Let A = [a,j] be an m x n matrix and let b be an m x 1 vector. Assume 
that z = (z1,Z2,...,Z,)? isa solution of the linear system Ax = b. Let 


B={j : 2; €0j}anda={i : aj #0 for some j € B}. 


Then Ax = b is sign-solvable iff the matrix [A[a, B] | — b[a]] is an S*-matrix and the matrix 
A(qa, B)? is an L-matrix. 

[KLM84] A matrix A is an L-matrix iff every row signing of A contains a unisigned column. 
[BCS94a] An m x n matrix A is a barely L -matrix iff 

(a) Ais an L-matrix. 


(b) For eachi = 1,2,...,n, there is a row signing of A such that column i is the only unisigned 
column. 

[BCS94a] An m x n matrix A is an S*-matrix iff n = m+ 1 and every row signing of A contains 

at least two unisigned columns. 

[BCS94a] An m x n matrix A is an S*-matrix iffn = m+ 1 and there exists a strict signing D such 

that AD and A(—D) are the only balanced column signings of A. 

[KLM84] A matrix A is an S-matrix iff A is an S*-matrix and every row of A is balanced. 

Let A be an m x n sign pattern that does not have a p x q zero submatrix for any positive integers 

p and q with p+ q > m. Then A is an L-matrix iff every strict row signing of A has a unisigned 

column. 


For further reading, see [Sha95], [Sha99], [KS00], and [SR04]. 


Examples: 


t+ - + 4+ 
+ + — +) isan L-matrix by Fact 12, and is a barely L-matrix by Fact 5 of Section 33.2. 
+ + + — 
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2. [BS95, p. 65] The m x (m+ 1) matrix 


-— + 0 0 0 

Sai sate 0 0 

- FS 0 0 O 
mel = 

+ 0 

+ 


is an S-matrix. Every strict column signing H/,,,D of H/,,, is an S*-matrix, with only two such 


strict column signings yielding S-matrices (namely, when D = +1). 


Applications: 


[BS95, Sec. 1.1] In supply and demand analysis in economics, linear systems, where the coefficients as well 
as the constants have prescribed signs, arise naturally. For instance, the sign-solvable linear system 


arises from the study of a market with one product, where the price and quantity are determined by the 
intersection of its supply and demand curves. 


33.4 Stability 


Definitions: 


A negative stable (respectively, negative semistable) real matrix is a square matrix B where each of the 
eigenvalues of B has negative (respectively, nonpositive) real part. In this section the term (semi)stable 
will mean negative (semi)stable. More information on matrix stability can be found in Section 9.5 and 
Chapter 19. 

A sign stable (respectively, sign semistable) sign pattern matrix is a square sign pattern A where every 
matrix B € Q(A) is stable (respectively, semistable). 

A potentially stable sign pattern matrix is a square sign pattern A where some matrix B € Q(A) isstable. 

Ann xn sign pattern matrix A allows a properly signed nest if there exists B € Q(A) anda permutation 
matrix P such that sgn(det(P? B P[{1, ... ,k}])) = (—D# for k= 1,...,n. 

A minimally potentially stable sign pattern matrix is a potentially stable, irreducible pattern such that 
replacing any nonzero entry by zero results in a pattern that is not potentially stable. 


Facts: 
Many of the following facts can be found in [BS95, Chap. 10]. 


1. A square sign pattern A is sign stable (respectively, sign semistable) iff each of the irreducible 
components of A is sign stable (respectively, sign semistable). 
2. [QR65] If A is ann x n irreducible sign pattern, then A is sign semistable iff 
(a) A has nonpositive main diagonal entries. 
(b) Ifi x j, then Aijaji S 0. 
(c) The digraph of A is a doubly directed tree. 
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3. If Aisann x n irreducible sign pattern, then A is sign stable iff 

(a) A has nonpositive main diagonal entries. 

(b) Ifi x j» then Ajjaji S 0. 

(c) The digraph of A is a doubly directed tree. 

(d) A does not have an identically zero determinant. 

(e) There does not exist a nonempty subset f of [1,2,... ,m] such that each diagonal element of 
A[B] is zero, each row of A[f] contains at least one nonzero entry, and no row of A[f, B] 
contains exactly one nonzero entry. 

The original version of this result was in terms of matchings and colorings in a graph ([JKD77, 
Theorem 2]); the restatement given here comes from [BS95, Theorem 10.2.2]. 

4. An efficient algorithm for determining whether a pattern is sign stable is given in [KD77], and the 
sign stable patterns have been characterized in finitely computable terms in [JKD87]. 

5. The characterization of the potentially stable patterns is a very difficult open question. 

6. The potentially stable tree sign patterns (see Section 33.5) for dimensions less than five are given 
in [JS89]. 

7. Ifann x n sign pattern A allows a properly signed nest, then A is potentially stable. 

8. In [JMO97], sufficient conditions are determined for an n x n zero—nonzero pattern to allow a 
nested sequence of nonzero principal minors, and a method is given to sign a pattern that meets 
these conditions so that it allows a properly signed nest. It is also shown that if A is a tree sign 
pattern that has exactly one nonzero diagonal entry, then A is potentially stable iff A allows a 
properly signed nest. 

9. In [LOD02], a measure of the relative distance to the unstable matrices for a stable matrix is defined 
and extended to a potentially stable sign pattern, and the minimally potentially stable patterns are 
studied. 

Examples: 
aa es Or F 0 Oy Be : 
1. [BS95] The pattern cE ‘| is sign stable, while the pattern ° ;| is sign semistable, but not 
sign stable. 
—1 1 0 
2. [JMO97] The matrix |—1 —1 1| has a (leading) properly signed nest, so that the pattern 
0 —3 1 
— + 0 
— — +} is potentially stable. 
0 -—- + 
—3 1 0 
3. [JMO97] The matrix B = 0 0 1| has —1 as a triple eigenvalue, and so is stable. Thus, 
8 —3 0 
the sign pattern A = sgn(B) is potentially stable, but it does not have a properly signed nest. 

4, [LOD02] The n x n tridiagonal sign pattern A with ay) = —,ajj4) = +,4j41; = — for i = 

1,...,— 1, and all other entries 0, is minimally potentially stable. 
Applications: 


[BS95, sec. 10.1] The theory of sign stability is very important in population biology ([Log92]). For 


instance, a general ecosystem consisting of n different populations can be modeled by a linear system 
of differential equations. The entries of the coefficient matrix of this linear system reflect the effects on 
the ecosystem due to a small perturbation. The signs of the entries of the coefficient matrix can often 
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be determined from general principles of ecology, while the actual magnitudes are difficult to determine 
and can only be approximated. The sign stability of the coefficient matrix determines the stability of an 
equilibrium state of the system. 


33.5 Other Eigenvalue Characterizations 
or Allowing Properties 


Definitions: 


A bipartite sign pattern matrix is a sign pattern matrix whose digraph is bipartite. 

A combinatorially symmetric sign pattern matrix is a square sign pattern A, where a;; 4 0 iffaj; A 0. 

The graph of a combinatorially symmetric n x n sign pattern matrix A = [q;;] is the graph with 
vertex set {1,2,...,}, where {i, j} is an edge iff aj; A 0. 

A tree sign pattern (t.s.p.) matrix is a combinatorially symmetric sign pattern matrix whose graph is a 
tree (possibly with loops). 

An n-cycle pattern is a sign pattern A where the digraph of A is an n-cycle. 

A k-consistent sign pattern matrix is a sign pattern A where every matrix B € Q(A) has exactly k real 
eigenvalues. 


Facts: 


1. [EJ91] Ann x n sign pattern A requires all real eigenvalues iff each irreducible component of A is 
a symmetric t.s.p. matrix. 
2. [EJ91] Ann x n sign pattern A requires all nonreal eigenvalues iff each irreducible component 
of A 
(a) Is bipartite. 


(b) Has all negative simple cycles. 
(c) Is SNS. 


3. [EJ93] For ann x n sign pattern A, the following statements are equivalent for each positive integer 
k>2: 
(a) The minimum algebraic multiplicity of the eigenvalue 0 occurring among matrices in Q(A) 
is k. 
(b) A requires an eigenvalue with algebraic multiplicity k, with k maximal. 


(c) The maximum composite cycle length in A is n — k. 


4. Ifan n x n sign pattern A does not require an eigenvalue with algebraic multiplicity 2 (namely, if 
A allows n distinct eigenvalues), then A allows diagonalizability. 

5. Sign patterns that require all distinct eigenvalues have many nice properties, such as requiring 
diagonalizability. In [LH02], a number of sufficient and/or necessary conditions for a sign pat- 
tern to require all distinct eigenvalues are established. Characterization of patterns that require 
diagonalizability is still open. 

6. [EHL94a] Ifa sign pattern A requires all distinct eigenvalues, then A is k-consistent for some k. 

7. [EHL94a] Let A be an n x n sign pattern and let A; denote the sign pattern obtained from A 
by replacing all the diagonal entries by +. Then A; requires n distinct real eigenvalues iff A is 
permutation similar to a symmetric irreducible tridiagonal sign pattern. 

8. [LHZ97] A3 x 3 nonnegative irreducible nonsymmetric sign pattern A allows normality iff AA? = 


0 
A’ Aand A is not permutation similar to |+ 0 +]. 
+ 
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9. 


10. 


11. 


12. 


13. 
14. 
15. 
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A; A : ; 
[LHZ97] If & } allows normality, where A is square, then the square pattern 
3 Ag 
A; A, Ad ... Ad 
Az; Ag Ag ... Ag 
Az; Ag Ag ... Ag 
Az; Ag Ag ... Ag 


also allows normality. Parallel results hold for allowing idempotence and for allowing nilpotence 
of index 2. (See [HL99] and [EL99].) 

[EL99] Suppose an n x n sign pattern A allows nilpotence. Then A” is compatible with 0, and for 
1<k<n-—1,tr(A*) is compatible with 0. Further, for each m,1 < m <n, E,,(A) (the sum of 
all principal minors of order m) is compatible with 0. 

[HL99] Let A be a5 x 5 irreducible symmetric sign pattern such that A? is compatible with A. 
Then A allows a symmetric idempotent matrix unless A can be obtained from the following by 
using permutation similarity and signature similarity: 


+ + 0 
++ - 0 - 
+ - + + 4+ 
+ 0+ 4+ - 
0 - + - 4 


[SG03] Let A be an n x n sign pattern. If the maximum composite cycle length in A is equal to the 
maximum rank (see section 33.6) of A, then A allows diagonalizability. 

[SG03] Every combinatorially symmetric sign pattern allows diagonalizability. 

A nonzero n x n(n > 2) sign pattern that requires nilpotence does not allow diagonalizability. 
Complete characterization of sign patterns that allow diagonalizability is still open. 


For further reading, see [Esc93a], [Yeh96], and [KMT96]. 


Examples: 


1. 


By Fact 1, the pattern 


* + 0 O 
aes RS SS BS 
0 — « O|° 
0 -— O x 


where each x entry could be 0, +, or —, requires all real eigenvalues. 


. [LH02] Up to equivalence (negation, transposition, permutational similarity, and signature sim- 


ilarity), the 3 x 3 irreducible sign patterns that require 3 distinct eigenvalues are the irreducible 
tridiagonal symmetric sign patterns, irreducible tridiagonal skew-symmetric sign patterns, and 
3-cycle sign patterns, together with the following: 


+ + 0] fo + o] fo + 0] fo -| [+ + 0 
0 0 +]/,J/- 0 +],/- 0 +],J/- 0 +],Jo0 0 + 
+ 0 0] |+ -— Of [+ 0 OF [+ - 0} [+ - 0 
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+ + 0 
3. [EL99] The sign pattern A = |— — —J] does not allow nilpotence, though it satisfies many 
—- + + 
“obvious” necessary conditions. 
+ t+ tee + 
Ae c= Or. 44k 28 
4. [LHZ97] Then x nsignpattern|, |, . | allows normality. 
a ae Ose DO 


33.6 Inertia, Minimum Rank 


Definitions: 


Let A bea sign pattern matrix. 

The minimal rank of A, denoted mr(A), is defined by mr(A) = min{ rank B: B € Q(A)}. 

The maximal rank of A, MR(A), is given by MR(A) = max{ rank B : B € Q(A) }. 

The term rank of A is the maximum number of nonzero entries of A no two of which are in the same 
row or same column. 

For a symmetric sign pattern A, smr(A), the symmetric minimal rank of A is 
smr(A) = min{ rank B: B = B’, B € Q(A)}. 

The symmetric maximal rank of A, SMR(A), is SMR(A) = max{ rank B: B = B’, B € Q(A)}. 

For a symmetric sign pattern A, the (symmetric) inertia set of A is in (A) = { in(B) : B = BT 
€ Q(A)}. 

A requires unique inertia if in(B,) = in(B2) for all symmetric matrices B,, By € Q(A). 

A sign pattern A of order n is an inertially arbitrary pattern (IAP) if every possible ordered triple 
(p,q,z) of nonnegative integers p,q, and z with p + q + z = n can be achieved as the inertia of some 
Be Q(A). 

A spectrally arbitrary pattern (SAP) isa sign pattern A of order n such that every monic real polynomial 
of degree n can be achieved as the characteristic polynomial of some matrix B € Q(A). 


Facts: 

1. MR(A) is equal to the term rank of A. 

2. Starting with a real matrix whose rank is mr(A) and changing one entry at a time to eventually 
reach a real matrix whose rank is MR( A), all ranks between mr( A) and MR(A) are achieved by real 
matrices. 

3. [HLW01] For every symmetric sign pattern A, MR(A)=SMR(A). 

4. [HS93] A sign pattern A requires a fixed rank r iff A is permutationally equivalent to a sign pattern 


of the form : is , where X isk x (r —k),0 <k <r, and Y and Z! are L-matrices. 


0 
5. [HLW01] A symmetric sign pattern A requires a unique inertia iff smr(A)=SMR(A). 
bse eh 0 A 
6. [HLW01] For the symmetric sign pattern A = At 5. of order n, we have 
i 


in(A) = {(k, k,n — 2k):mr(A,) < k < MR(A))}. 


In particular, 2 mr(A;) = smr(A). 

7. [DJO00], [EOD03] Let T,, be the n x n tridiagonal sign pattern with each superdiagonal entry 
positive, each subdiagonal entry negative, the (1,1) entry negative, the (n,n) entry positive, and 
every other entry zero. It is conjectured that T,, is an SAP for all n > 2. It is shown that for 
3 <n < 16, T, isan SAP. 
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8. [GSO1] Let S, be the n x n(n > 2) sign pattern with each strictly upper (resp., lower) triangular 
entry positive (resp., negative), the (1,1) entry negative, the (n,n) entry positive, and all other 
diagonal entries zero. Then S,, is inertially arbitrary. 

MLO02] Further, if the (1, 7) and (n, 1) entries of S,, are replaced by zero, the resulting sign pattern 
is also an IAP. 

9. [CV05] Not every inertially arbitrary sign pattern is spectrally arbitrary. 
10. [MOT03] Suppose 1 < p < n — 1. Then every n x n sign pattern with p positive columns and 
n — p negative columns is spectrally arbitrary. 


For further reading see [CHL03], [SSG04], [Hog05], and references [Nyl96], [JD99], [BFH04], [BF04], 
and [BHL04] in Chapter 34. 


Examples: 
1. [HLWO1] Let 


+ 0 + + 

Oo + + + 
A= 

+ + -— 0 

+ + 0 - 


Then in(A) = (2, 2,0), smr(A) = SMR(A) = 4, but 3 = mr(A) < smr(A) = 4. 
2. [HLO1] Let J, be the n x n sign pattern with all entries equal to +. Then 


inJ,) = {(s,t,n-—s—t):s >1,t>0,s+t <n}. 


3. [Gao01], [GS03], [HLO1] Let A be the n x n (nm > 3) sign pattern all of whose diagonal entries are 
zero and all of whose off-diagonal entries are +. Then 


in(A) = {(s,t,n —s —t):s > 1,t>2,s5+t <n}. 


33.7 Patterns That Allow Certain Types of Inverses 


Definitions: 


A sign pattern matrix A is nonnegative (positive), denoted A > 0(A > 0), if all of its entries are 
nonnegative (positive). 

An inverse nonnegative (inverse positive) sign pattern matrix is a square sign pattern A that allows an 
entrywise nonnegative (positive) inverse. 

Let both B and X be real matrices or nonnegative sign pattern matrices. Consider the following con- 
ditions. 


(1) BXB=B. 


(2) 
(3) BX is symmetric. 
(4) XB is symmetric. 


For a real matrix B, there is a unique matrix X satisfying all four conditions above and it is called the 
Moore-Penrose inverse of B, and denoted by B More generally, let B{i, j,...,1} denote the set of matrices 
X satisfying conditions (7), (j),..., (1) from among conditions (1)—(4). A matrix X € B{i,j,...,J} is 
called an (i, j,...,1)-inverse of B. For example, if (1) holds, X is called a (1)-inverse of B; if (1) and (2) 
hold, X is called a (1, 2)-inverse of B, and so forth. See Section 5.7. 
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For a nonnegative sign pattern matrix B, if there is a nonnegative sign pattern X satisfying (1) to (4), 
then X is unique and it is called the Moore—Penrose inverse of B. An (i, j,...,/)-inverse of B is defined 
similarly as in the preceding paragraph. 


Facts: 
The first three facts below are contained in [BS95, Chap. 9]. 


1. [JLR79] Let A be an n x n (+,—)-pattern, where n > 2. Then A is inverse positive iff A is not 

An Ap 

Az Az2 
blocks Aj;, Az. are square or rectangular, and one (but not both) of Aj;, A2. may be empty. 

2. The above result in [JLR79] is generalized in [FG81] to (+, —, 0)-patterns, and additional equiva- 
lent conditions are established. Let e denote the column vector of all ones, J the n x n matrix all 
of whose entries equal 1, and A’ the matrix obtained from A by replacing all negative entries with 
zeros. For ann x n fully indecomposable sign pattern A, the following are equivalent: 


permutationally equivalent to a pattern of the form , where Aj. < 0, Az; > 0, the 


(a) A is inverse positive. 
(b) A is not permutationally equivalent to a pattern of the form 
An Ap 
An, Ax 
(but not both) of Aj), Ax. may be empty. 


, where Aj) < 0, Ao; > 0, the blocks Aj;, Az: are square or rectangular, and one 


is irreducible. 


A 
(c) The pattern gts Gj 


(d) There exists B € Q(A) such that Be = Ble = 0. 
(e) There exists a doubly stochastic matrix D such that (D — 4 ) € Q(A). 
3. [Joh83] For ann x n fully indecomposable sign pattern A, the following are equivalent: A is inverse 


nonnegative; A is inverse positive; — A is inverse nonnegative; — A is inverse positive. 
4. [EHL97] Ifann x n sign pattern A allows B and B™! to be in Q(A), then 


(a) MR(A) = n. 
(b) A? is compatible with I. 


(c) adj A is compatible with A and det(A) is compatible with +, or, adj A is compatible with — A 
and det(A) is compatible with —, where adj A is the adjoint of A. 


5. In [EHL94b], the class G of all square patterns A that allow B,C € Q(A) where BCB = B is 
investigated; it is shown for nonnegative patterns that G coincides with the class of square patterns 
that allow B € Q(A) where B* = B. 

6. [HLR04] An m x n nonnegative sign pattern A has a nonnegative (1, 3)-inverse (Moore—Penrose 
inverse) iff A allows a nonnegative (1, 3)-inverse (Moore—Penrose inverse). 


For further reading, see [BF87], [BS95, Theorem 9.2.6], [SSO1], and [SS02]. 


Examples: 
1. By Facts 2 and 3, the sign pattern 


Oo + + 
O + + + 
A= 

ee wes gn OG 
aa! | as 


is not inverse nonnegative. 
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2. [EHL97] The sign pattern 


+ + + 4+ 

0 + + + 
A= 

(i oe 

0 -— + + 


satisfies all the necessary conditions in Fact 4, but it does not allow an inverse pair B and B~! in 


Q(A). 


33.8 Complex Sign Patterns and Ray Patterns 


Definitions: 
A complex sign pattern matrix is a matrix of the form A = A, +7A) for some m x n sign patterns A, 
and A), and the sign pattern class or qualitative class of A is 


Q(A) = {Bi +7B : By, € Q(A;) and Bz € Q(A3)}. 


Many definitions for sign patterns, such as SNS, extend in the obvious way to complex sign patterns. 
The determinantal region of a complex sign pattern A is the set 


Sa = {det(B) : B € Q(A)}. 


A ray pattern is a matrix each of whose entries is either 0 or a ray in the complex plane of the form 
{re!® : r > 0} (which is represented by e’”). The ray pattern class of an m x n ray pattern A is 


Q(A) = {B = [Bq] € Minxn(C) : Bpq = 0 iffapq = 0, and otherwise arg bp, = arg apq}. 


For a < , the open sector from the ray e’” to the ray e’P is the set of rays {rel? : r > 0,a <O < B}. 
The determinantal region of a ray pattern A is the set 


Ra = {det(B) : B € Q(A)}. 


Ann x n ray pattern A is ray nonsingular if the Hadamard product X o A is nonsingular for every 
entrywise positive m x n matrix X. 
A cyclically real ray pattern is a square ray pattern A where the actual products of every cycle in A is 
real. 


Facts: 

1. [EHL98], [SSO5] For a complex sign pattern A, the boundaries of S, are always on the axes on the 
complex plane. 

2. [EHL98], [SS05] For a sign nonsingular complex sign pattern A, S, is either entirely contained in an 
axis of the complex plane or is an open sector in the complex plane with boundary rays on the axes. 

3. [SS05] For a complex sign pattern or ray pattern A, the region S4\{0} (or R4\{0}) is an open set 
(in fact, a disjoint union of open sectors) in the complex plane, except in the cases that S4 (Ra) is 
entirely contained in a line through the origin. 

4. The results of [MOT97], [LMS00], and [LMS04] show that there is an entrywise nonzero ray 
nonsingular ray pattern of order n if and only if 1 < n < 4. 

5. [EHL00] An irreducible ray pattern A is cyclically real iff A is diagonally similar to a real sign 
pattern. More generally, a ray pattern A is diagonally similar to a real sign pattern iff Aand A+ A* 
are both cyclically real. 
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Examples: 
1. [EHL98] If A = i a +1 _|> then A is sign nonsingular and S, is the open sector from 
the ray e~'”/? to the ray e'”/?. 
eit /2 oh 
+ ete 4 . . 
2. [MOT97] The ray pattern eit/2 is ray nonsingular. 
LL ae eit /2 
3. [EHLOO] Let 
0 et 9 —e ih 
0 + —e i 9 
A= eit) _pih 9g — eis > 
eff - 0 - 


where 6; and 6) are arbitrary. Then A is cyclically real, and A is diagonally similar (via the diagonal 
ray pattern S = diag(+, e’, e+), —e% ) to 


0 + 0 + 
0 + - 0 
+ —- 0 + 
ee ee | 


33.9 Powers of Sign Patterns and Ray Patterns 


Definitions: 


Let J, (or simply J ) denote the all + sign (ray) pattern of order n. 

A square sign pattern or ray pattern A is powerful if all the powers A', A”, A>,..., are unambiguously 
defined, that is, no entry in any A* isa sum involving two or more distinct rays. For a powerful pattern A, 
the smallest positive integers ] = 1(A) and p = p(A) such that A! = A!*? are called the base and period 
of A, respectively. 

A square sign pattern or ray pattern A is k-potent if k is the smallest positive integer such that A = A**!, 


Facts: 


1. [LHE94] An irreducible sign pattern A with index of imprimitivity h (see Section 29.7) is powerful 
iff all cycles of A with lengths odd multiples of h have the same sign and all cycles (if any) of A 
with lengths even multiples of h are positive (see [SS04]). A sign pattern A is powerful iff for every 
positive integer d and for every pair of matrices B,C € Q(A), sgn(B“) = sgn(C*). 

2. [LHE94] Let A be an irreducible powerful sign pattern, with index of imprimitivity h. Then the 
base and the period of A are given by /(A) = I(|A]), p(A) = h if A does not have any negative 
cycles, and p(A) = 2h if A has a negative cycle. 

3. [Esc93b] The only irreducible idempotent sign pattern of order n > 2 is the all + sign pattern. 

4, [LHE94], [SEK99], [SBS02] Every k-potent irreducible sign or ray pattern matrix is powerful. 

5, [EL97] The maximum number of — entries in the square of a sign pattern of order n is n* — 2; the 
maximum number of — entries in the square of a (+, —) sign pattern of order n is [n?/2]. 

6. [HLO1b] Let A be an n x n (n > 3) sign pattern. If A has only one entry that is not nonpositive, 
then A* has at most n* — n negative entries. 

7. [LHS02] Let A be an irreducible ray pattern. Then A is powerful iff A is diagonally similar to a 
subpattern of e'“J for some a € R, where J is the all + ray pattern. 
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8. [LHS05] Suppose that A = a a is a powerful ray pattern, where A,; (resp., Az) is 
22 
irreducible with index of imprimitivity h, (resp., hy) and 0 4 Ay X C1Jn,,0 A An KX C2Jy,. If 
Iem(hy, hz) 
Ai2 # 0, then (2) =1 


For further reading, the structures ofk-potent sign patterns or ray patterns are studied in [SEK99], 
[Stu99], [SBS02], [LG01], and [Stu03]. 


Examples: Os ho ae: oe = s 
0 0 () Oo + 

1. [LHE94] The reducible sign pattern A = ; satisfies A? = 5 4 i and 
0 0 0 0 0 0 0 0 


A? = A. Thus, A is 2-potent and yet A is not powerful. 
2. [SEK99] Let P,, be the n x n circulant permutation sign pattern with (1, 2) entry equal to +. Let 
Q,, be the sign pattern obtained from P,, by replacing the + in the (n, 1) position with a —. Then 
P,, is n-potent and Q,, is 2n-potent. 
0 J pxq 0 
3. [SBSO2] Suppose that 3|k. Let A= mw] 0 0 Jase | 4 where Jinx; denotes the all ones 
Jrxp 0 0 


m X n matrix and @° is a primitive k/3-th root of unity. Then A is a k-potent ray pattern. 


33.10 Orthogonality 


Definitions: 
A square sign pattern A is potentially orthogonal (PO) if A allows an orthogonal matrix. 

A square sign pattern A that does not have a zero row or zero column is sign potentially orthogonal 
(SPO) if every pair of rows and every pair of columns allows orthogonality. 


Two vectors x = [x),...,x,] andy = [y,..., ¥,] are combinatorially orthogonal if 
I{i:xiyi AO} Al. 
Facts: 
1. Every PO sign pattern is SPO. 
2. [BS94], [Wat96] For n < 4, every n x n SPO sign pattern is PO. 
3. [Wat96] There is a5 x 5 fully indecomposable SPO sign pattern that is not PO. 
4. [JW98] There is a6 x 6 (+, —) sign pattern that is SPO but not PO. 
5. [BBS93] Let A be an n x n fully indecomposable sign pattern whose rows are combinatorially 


orthogonal and whose columns are combinatorially orthogonal. Then A has at least 4(n — 1) 
nonzero entries. This implies that a conjecture of Fiedler [Fie64], which says a fully indecomposable 
orthogonal matrix of order n has at least 4(m — 1) nonzero entries, is true. 

6. [CJL99] For n > 2, there is an n x n fully indecomposable orthogonal matrix with k zero entries 
iff0 <k < (n—2)?. 

7. [EHH99] Let S be any skew symmetric sign pattern of order n all of whose off-diagonal entries are 
nonzero. Then I + S is PO. 

8. [EHH99] It is an open question as to whether every sign pattern A that allows an inverse in Q(A‘) 
is PO. 
For further reading see [Lim93], [Sha98], [CS99], and [CHR03]. 
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Examples: 


1. [BS94] Every 3 x 3 + SPO sign pattern can be obtained from the following sign pattern by using 
permutation equivalence and multiplication by signature patterns: 


+ 
+ - + 
2. [Wat96] The sign pattern 

Se ee FQ: 

t 0 
O+ +++ 
oe ee ee 
-- - ++ 


is SPO but not PO. 


33.11 Sign-Central Patterns 


Definitions: 


A real matrix B is central if the zero vector is in the convex hull of the columns of B. A real or sign pattern 
matrix A is sign-central if A requires centrality. 

A minimal sign-central matrix A is a sign-central matrix that is not sign-central if any column of A is 
deleted. 

A tight sign-central matrix is a sign-central matrix A for which the Hadamard (entrywise) product of 
any two columns of A contains a negative component. 

A nearly sign-central matrix is a matrix that is not sign-central but can be augmented to a sign-central 
matrix by adjoining a column. 


Facts: 


1. [AB94] An m x n matrix A is sign-central iff the matrix DA has a nonnegative column vector for 
every strict signing D of order m. 

2. [HKK03] Every tight sign-central matrix is a minimal sign-central matrix. 

3. [LCO00] If A is nearly sign-central and [A | a] is sign-central, then [A | a] is also sign-central for 
every a’ # 0 obtained from a by zeroing out some of its entries. 


For further reading, see [BS95, Sect. 5.4], [DD90], [LLS97], and [BJS98]. 


Examples: 


1. [BS95, p. 100], [HKK03] For each positive integer m, the m x 2” + sign pattern E,,, such that each 
m-tuple of +’s and —’s is a column of E,,,, is a tight sign-central sign pattern. 
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This chapter assumes basic terminology from graph theory in Chapter 28; a good general graph theory 
reference is [CL96]. For standard terms or concepts from matrix analysis, see Part 1: Basic Linear Algebra, 
particularly Chapter 4.3, and Chapter 8; a good general matrix reference is [HJ85]. As we will be interested 
in properties of A that are permutation similarity invariant, primarily eigenvalues and their multiplicities, 
we will generally view a graph as unlabeled, except when referencing by labels is convenient. 

For a given simple graph G on n vertices, let S(G) (respectively, H(G)) denote the set of all nxn real 
symmetric (respectively, complex Hermitian) nxn matrices A = [a;;] such that fori 4 j, aj; A 0 ifand 
only if there is an edge between i and j. 

Our primary interest lies in the following very general question. Given G, what are all the possible lists 
of multiplicities for the eigenvalues that occur among matrices in S(G) (respectively, H(G))? Much of 
our focus here is on the case in which G = T is a tree. 

It is important to distinguish two possible interpretations of “multiplicity list.” Since the eigenvalues 
of a real symmetric or complex Hermitian matrix are real numbers, they may be placed in numerical 
order. If the multiplicities are placed in an order corresponding to the numerical order of the underlying 
eigenvalues, then we refer to such a way of listing the multiplicities as ordered multiplicities. If, alternatively, 
the multiplicities are simply listed in nonincreasing order of the values of the multiplicities themselves, we 
refer to such a list as unordered multiplicities. For example, if A has eigenvalues —3, 0,0, 1, 2, 2, 2,5, 7, the 
list of ordered multiplicities is (1, 2, 1, 3, 1, 1), while the list of unordered multiplicities is (3, 2, 1, 1, 1,1). In 
either case, such a list means that there are exactly 6 different eigenvalues, of which 4 have multiplicity 1. 
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Ifa graph G is not connected, then the multiplicity lists for G may be deduced from those of its compo- 
nents via superposition. Also, graphs with many edges admit particularly rich collections of multiplicity 
lists. For example, the complete graph admits all multiplicity lists with the given number of eigenvalues, 
except the list in which all eigenvalues are the same. For these reasons, a natural beginning for the study 
of multiplicity lists for S(G) or H(G) is the case in which G = T, a tree. In addition, trees present several 
attractive features for this problem, so much of the research in this area, and this chapter, focuses on trees. 


34.1 Multiplicities and Parter Vertices 


Definitions: 


Let G bea simple graph. 

For an Xn real symmetric or complex Hermitian matrix A = [a;;], the graph of A, denoted by G(A), 
is the simple graph on n vertices, labeled 1,2, ... ,n, with an edge between i and j if and only if a;; # 0. 

Let S(G) (respectively, 74(G)), denote the set of all nxn real symmetric (respectively, complex Hermi- 
tian) matrices A such that G(A) = G (where G has n vertices). No restriction is placed upon the diagonal 
entries of A by G, except that they are real. 

If G’ is the subgraph of G induced by 8, A(G’) can be used to denote A(f) and A[G’] to denote A[]. 

Given a tree T, the components of T \ {7} are called branches of T at j. 

If A € H(G), A € o(A), and j is an index such that a4(;)(4) = o4(A) + 1, then j is called a Parter 
index or Parter vertex (for A, A and G) (where w,4(A) denotes the multiplicity of 1). Some authors refer 
to such a vertex as a Parter—Wiener vertex or a Wiener vertex. 

If j is a Parter vertex for an eigenvalue A of an A € 1(G) such that A occurs as an eigenvalue of at least 
three direct summands of A(j), j is called a strong Parter vertex. 

A downer vertex i in a graph G (for 4 € o(A) and A € H(G)) is a vertex i such that w4(j)(A) = 
a A(a) -—1. 

A downer branch of a tree T at j is a branch T; at j, determined by a neighbor i of j such that 7 is a 
downer vertex in T; (for A and A[T;]). 

If a branch of a tree at a vertex j is a path and the neighbor of j in this branch is a pendant vertex of 
this path, the branch is a pendant path at j. 


Facts: 


1. If A € H(G), then trivially A(@i) € H(G \ {i}). 
2. [HJ85] (Interlacing Inequalities) Ifan nxn Hermitian matrix A has eigenvalues, < 42 <-+-- < An 
and A(i) has eigenvalues 6,1 < Bi2 < +--+ S Bin-1, then dA, < Bir < Ao S Biz S++ S Bin 


ae er inl Eee, 
3. If X is an eigenvalue of an Hermitian matrix A, then a4(A) — 1 < a@aqy)(A) < a@a(A) + 1 for 
fe eae 


4. If T isa tree, then any matrix of H(T) is diagonally unitarily similar to one in S(T). 
5. [JLS03a](Parter-Wiener Theorem: Generalization) Let T be a tree and A € S(T). Suppose that 
there exists an index i and a real number A such that  € 0(A) No (A(i)). Then 


¢ There is in T a Parter vertex j for A. 


° Ifa@,(A) > 2, then j may be chosen so that 57(j) > 3 and so that there are at least three 
components 7), 75, and T3 of T \ {j} such that w4j7)(A) > 1,k = 1,2,3. 

° Ifa,(A) = 1, then j may be chosen so that there are two components T, and T, of T\{j} such 
that al 7] (A) = 1, k= M2: 


6. [JLS03a] For A € S(T), T a tree, j is a Parter vertex for A if and only if there is a downer branch 
at j fora. 
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7. JL] Suppose that G is a simple graph on n vertices that is not a tree. Then 


* There is a matrix A € S(G) with an eigenvalue 4 such that there is an index j so that a4(A) = 
aacj)(A) = Land aay(A) < 1 for everyi = 1,...,n. 


* There is a matrix B € S(G) with an eigenvalue A such that ag (A) > 2 andagy)(A) = ag(A)— 1, 
for everyi = 1,...,”. 


8. [JLSSW03] Let T bea tree and A, < Az be eigenvalues of A € S(T) that share a Parter vertex in T. 
Then there is at least one A € o (A) such that A, < A < Ap. 
9. Let T beatree and A € S(T). If T’ is a pendant path of a vertex j of T, then T’ is a downer branch 
for each eigenvalue of the direct summand A[T’]. 
10. Let T bea path and A € S(T). Then each eigenvalue of A has multiplicity 1. 
11. Let A be an nxn irreducible real symmetric tridiagonal matrix. Then 


* Ahas distinct eigenvalues. 
* In A(z), there are at most min{i — 1, — i} interlacing equalities and this number may occur. 


* For each interlacing equality that does occur, the relevant eigenvalues must be an eigenvalue (of 
multiplicity 1) of both irreducible principal submatrices of A(7). 


See Example 3 below. 


Examples: 


1. Ingeneral, one might expect that in passing from A to A(z), multiplicities typically decline. However, 
Fact 5 is counter to this intuition in the case for trees. A rather complete statement has evolved 
through a series of papers ([Par60], [Wie84], [JLS03a]). In particular, Fact 5 says that when T is 
a tree and a,4(A) > 2, there must be a strong Parter vertex because, by interlacing, the hypothesis 
A € o(A)No(A(z)) must be satisfied for any i. However, i itself need not be a Parter vertex. Even 
when a4(A) > 2, it can happen that a(j)(A) = o4(A) + 1 with 67(j) = 1 or 67(j) = 2 0rd 
appears in only one or two components of T \ {j}, even if57(j) > 3. There may, as well, be several 
Parter vertices and even several strong Parter vertices. Much information about Parter vertices may 
be found in [JLSSW03] and [JLS03a]. Let A, 4 € R, A ¥ p, and consider real symmetric matrices 
whose graphs are the following trees, assuming that every diagonal entry corresponds to the label 
of the corresponding vertex. 


The vertex v is a Parter vertex for 4 in real symmetric matrices for which the graph is each of 
the trees in Figure 34.1. We also note that, depending on the tree T, several different vertices 
of T could be Parter for an eigenvalue of the same matrix in S(T). The matrices A[T,] and 
A[T)] each have u and v as Parter vertices for A. 


T3 


FIGURE 34.1 Examples of Parter vertices. 
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* Also, depending on the tree T, the same vertex could be 
a Parter vertex for different eigenvalues of a matrix in 
S(T). The vertex v is a Parter vertex for A and pu ina real 
symmetric matrix A for which the graph is the tree in 
Figure 34.2. Such a matrix A has A and y as eigenvalues 
with a4(A) = 2 = aa(y). Since it is clear that we have pIGURE34.2. Vertex v is a Parter 
aav)(A) = 3 = aay) (2), it means that v is Parter for vertex for A and pu. 
and pw. 


Though a notion of “Parter vertex” can be defined for nontrees, Fact 7 is the converse to Fact 5 that 
shows that its remarkable conclusions are generally valid only for trees. 

Consider the matrix J3 whose graph is the cycle C3 (the possible multiplicities for the eigenvalues 
of a matrix whose graph is a cycle was studied in [Fer80]), which is not a tree. The matrix J; has 
eigenvalues 0, 0,3. Since the removal of any vertex from C3 leaves a path, we conclude that there is 
no Parter vertex for the multiple eigenvalue 0. 

If a graph G is a path on n vertices, then G is a tree and if the vertices are labeled consecutively, 
any matrix in S(G) is an irreducible tridiagonal matrix. Conversely, the graph of an irreducible 
real symmetric tridiagonal matrix is a path. The very special spectral structure of such matrices has 
been of interest for some time for a variety of reasons. Two well-known classical facts are that all 
eigenvalues are distinct (i-e., all 1s is the only multiplicity list) and, if a pendant vertex is deleted, 
the interlacing inequalities are strict. Both statements follow from Fact 5, but more can be gotten 
from Fact 5 as well. If A is n x n real symmetric and 1 < i < n, then as many as n — 1 of the 
eigenvalues of A(i) might coincide with some eigenvalue of A. We refer to such an occurrence as 
an “interlacing equality.” If a pendant vertex is removed from a path, no interlacing equalities can 
occur, but if an interior vertex is removed, interlacing equalities can occur. The complete picture 
in this regard may be also be deduced from Fact 5. 


Maximum Multiplicity and Minimum Rank 


Definitions: 


Let G be a simple graph. 


The 


maximum multiplicity of G, M(G), is the maximum multiplicity for a single eigenvalue among 


matrices in S(G). 


The 
The 


minimum rank of G is mr(G) = minyesig) rank(A). 
path cover number of G, P(G), is the minimum number of induced paths of G that do not 


intersect, but do cover all vertices of G. 
A(T) = max[p — q] over all ways in which q vertices may be deleted from G, so as to leave p paths. 
Isolated vertices count as (degenerate) paths. 


The 


Facts: 


maximum rank deficiency for G is m(G) = n — mr(G), where the order of G is n. 


Let G be a simple connected graph of order n. 


1. If G isa path, P(G) = 1; otherwise P(G) > 1. 


There may be many minimum path covers. See Example 2. 
Maximizing sets of removed vertices (used in the computation of A(G)) are not unique; even the 
number of removed vertices is not unique. See Example 3. 
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FIGURE 34.3 Two of the forbidden graphs for mr(G) = 2. 


[Fie69] M(G) = 1 ifand only if G isa path. 

M(G) =n — 1ifand only if G is the complete graph K,,. 

M(G)=m(G). 

[JL99] For a tree T, M(T) = A(T) = P(T) = m(T). 

[JSO2] For any tree T, let Hy denote the subgraph of T induced by the vertices of degree at least 3. 
For a tree T, A(T) can be computed by the following algorithm. See Example 4. 


Algorithm 1: Computation of A(T) 
Given a tree T. 


1. Set Q=@and T’ =T. 


2. While Hr 4 @: 
Remove from T” all vertices v of Hy such that dp(v) — 6x,,(v) > 2 
and add these vertices to Q. 


3. A(T) = p — |Q| where p is the number of components (all of which are paths) in T\Q. 


[JL99], [BFH04] A(G) < M(G) and A(G) < P(G). 


. [BFH04], [BFH05] Ifn > 2, P(K,) < M(K,,), where K,, is the complete graph on n vertices. If G 


is unicyclic (ie., has a unique cycle), then P(G) > M(G) and strict inequality is possible. 
[BFH04] Minimum rank (and, thus, maximum multiplicity) of a graph with a cut vertex can be 
computed from the minimum ranks of induced subgraphs. 

[BHL04] If H is an induced subgraph of G, then mr(H) < mr(G). Furthermore, mr(G) =2 
(Le., M(G) = n— 2) ifand only if G does not contain as an induced subgraph one of the following 
four forbidden graphs: the path on 4 vertices P4, the complete tripartite graph K333, the two graphs 
shown in Figure 34.3. Other characterizations are also given. 


Examples: 


1. 


Considering the tree T in Figure 34.4, we have P(T) = 2 
(e.g., 1-3-2 and 5-4-6 constitute a minimal path cover of the 
vertices) and, of course, A(T) = 2, as removal of vertex 4 
leaves the 3 paths 1-3-2, 5, and 6 (and neither can be improved 
upon). Note that if submatrices A[{1, 2, 3}], A[{5}], and A[{6}] 
of A € S(T) are constructed so that A is an eigenvalue of each 
(this is always possible and no higher multiplicityin any ofthem FIGURE34.4_ Atreewith path cover 
is possible), then a4(A) > 3 — 1 = 2, whichis the maximum  number?. 

possible. 

Consider the tree T on 12 vertices in Figure 34.5. It is not difficult to see that the path cover number 
of T, P(T), is 4. However, it can be achieved by different collections of paths. For example, P(T) 
can be achieved from the collection of 4 paths of T, 1-2-3, 4-5-6, 7-8 and 12-9-10-11. Similarly, 
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FIGURE 34.5 A tree with path cover number 4. 


the paths of T, 1-2-3, 4-5-8-7, 6 and 12-9-10-11, form a collection of vertex disjoint paths (each 
one is an induced subgraph of T) that cover all the vertices of T. 

3. Consider the tree T on 12 vertices in Figure 34.5. As we can see in Table 34.1, A(T) can be achieved 
for q = 1,2,3. When q = 2, there are 3 different sets of vertices whose removal from T leaves 6 
components (paths), i.e., p — q = 4. 

4. The algorithm in Fact 8 applied to the tree T in Figure 34.6 gives, in one step, a subset of vertices 
of T, Q = {v2, V3, V4, V5}, with cardinality 4 and such that T\Q has 13 components, each of which 
is a path. Therefore, A(T) = 13-4=9. 

Note that, in any stage of the process to determine A(T), we may not choose just any vertex with 
degree greater than or equal to 3. 


TABLE 34.1 A(T) for the tree T in Figure 34.5 


Removed VerticesfromT q p p—q(=A(T)) 

5 1 5 4 
5,2 2 6 4 
5,9 2 6 4 
5,10 2 6 4 
2,5,9 3 7 4 
2,5, 10 3 7 4 

Us 

U4 V1 Vo 
U3 


FIGURE 34.6 A tree T with A(T) = 9. 
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FIGURE 34.7 A tree of diameter 6 for which the minimum number of distinct eigenvalues is 8 > 6 + 1. 


34.3. The Minimum Number of Distinct Eigenvalues 


Definitions: 
Let T bea tree. 

The diameter of T, d(T), is the maximum number of edges in a path occurring as an induced subgraph 
of T. 

The minimum number of distinct eigenvalues of T, N’(T), is the minimum, among A € S(T), 
number of distinct eigenvalues of A. 


Facts: 


1. [JLO2a] Let T bea tree. Then V(T) > d(T) +1. 
2. [JSa2] If T is a tree such that d(T) < 5, then there exist matrices in S(T) attaining as few distinct 
eigenvalues as d(T) + 1. 


Examples: 


1. Since each entry in a multiplicity list represents a distinct eigenvalue, the “length” ofa list represents 
the number of different eigenvalues. This number can be as large as n (the number of vertices), of 
course, but it cannot be too small. Restrictions upon length limit the possible multiplicity lists. Just 
as a path has many distinct eigenvalues, a long (chordless) path occurring as an induced subgraph 
of a tree forces a large number of distinct eigenvalues. 

2. For many trees T, there exist matrices in S(T) attaining as few distinct eigenvalues as d(T) + 1. 
However, for the tree T in Figure 34.7, d(T) = 6 and, in [BF04], the authors have shown that 
N(T) = 8 > d(T)+1. Itis not known how to deduce the minimum number of distinct eigenvalues 
from the structure of the tree, in general. 


34.4 The Number of Eigenvalues Having Multiplicity 1 


Definitions: 


Given a tree T, let /(T) be the minimum number of 1s among multiplicity lists occurring for T. 


Facts: 


1. [JLSO3a] For any tree T, the largest and smallest eigenvalues of any A € S(T) necessarily have 
multiplicity 1. 

2. [JLS03a] For any tree T on n > 2 vertices, U(T) > 2 and, for each n, there exist trees T for which 
Uu(T) =2. 

3. Let T bea tree on n vertices. U(T) > 2N(T) — n. In particular, /(T) > 2(d(T) + 1) — n. 
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Examples: 


1. As with the length of lists, it is relatively easy to have many 
1s in a multiplicity list. The more interesting issue is how few of 
1s may occur among lists for a given tree T’. It certainly depends 
upon the tree, as the star (see Figure 34.8) may have just two 1s, 
while a path (see Figure 34.9) always has as many as the number 
of vertices. FIGURE 34.8 A star. 

2. If T has a diameter that is large relative to its number of vertices (a path is an extremal example), 
then it may have to have a minimum number of distinct eigenvalues, which forces //(T) to be much 
greater than 2. However, U/(T) may be greater than 2 for other reasons. For example, for the tree T 
in Figure 34.10, d(T) = 4, n = 8, but U(T) = 3. It is not known how U/(T) is determined by T, 
and it appears to be quite subtle. 


34.5 Existence/Construction of Trees with Given Multiplicities 


Definitions: 
For a given graph G, the collection of all multiplicity lists is denoted by £(G). If it is not clear from the 
context, we will distinguish the unordered lists as £,,(G) from the ordered lists £,(G). 

General Inverse Eigenvalue Problem (GIEP) for S(T): Given a vertex v of a tree T, what are all the 
sequences of real numbers that may occur as eigenvalues of A and A(v), as A runs over S(T)? 

Inverse Eigenvalue Problem (IEP) for S(T): What are all possible spectra that occur among matrices 
in S(T), T being a tree? 

A tree T has equivalence of the ordered multiplicity lists and the IEP if a spectrum occurs for some 
matrix in S(T) whenever it is consistent with some list of ordered multiplicities of £,(T). 


Facts: 


1. [Lea89] Let T bea tree on n vertices and v bea vertex of T. Let d),A2,... 5A, and [11, [2,.-- » Un—1 
be real numbers. If 


Ay < fy < Aq < +++ < Mn- < An 


then there exists a matrix A in S(T) with eigenvalues A,,A2,...,A,, and such that, A(v) has 


eigenvalues /1j, (125... 5 n—1- 
2. For any tree T on n vertices and any given sequence of n distinct real numbers, there exists a matrix 


O-O—O-O0-0-0-O-: -- 


FIGURE 34.9 A path. 


FIGURE 34.10 A tree T on 8 vertices with d(T) = 4andU(T) = 3. 
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FIGURE 34.11 A tree for which there is not equivalence between ordered multiplicity lists and the IEP. 


in S(T) having these numbers as eigenvalues. 

3. Any path has equivalence of the ordered multiplicity lists and the IEP. 

4, [BF04] There exists a tree for which the equivalence of the ordered multiplicity lists and the IEP is 
not verified. (See Example 1.) 


Examples: 


1. For remarkably many small trees and many of the families of trees to be discussed in the next 
section, the ordered multiplicity lists are equivalent to the IEP. This is not always the case. Extremal 
multiplicity lists for large numbers of vertices can force numerical relations upon the eigenvalues, 
as in the tree T’ shown in Figure 34.11. 

Let A be the adjacency matrix of T (i.e., the 0,1-matrix in S(T) with all diagonal entries 0). The 
Parter—-Wiener Theorem (Fact 5 Section 34.1) guarantees that a 4(0) = 4, and likewise guaran- 
tees two eigenvalues of multiplicity 2 (the nonzero eigenvalues of A[{2,3,4}] = A[{5,6, 7}] = 
AL[{8, 9, 10}], so the ordered multiplicity list of A is (1, 2, 4, 2, 1). In fact, direct computation shows 
that o(A) = (—/5, -V2, —V2, 0,0, 0, 0, V2, V2, /5). 

However, it is not possible to prescribe arbitrary real numbers as the eigenvalues with this ordered 
multiplicity list. If A = [a;;] € S(T) has eigenvalues A, < Az < A3 < Aq < As with multiplicities 
1, 2, 4, 2, 1, respectively, then A; +A5 = A2+A4. The method used to establish this restriction comes 
from [BF04]. By the Parter-Wiener Theorem and an examination of subsets of vertices, aj, = A3. 
The restriction then follows from comparison of the traces of A and A(1). 

It is not known for which trees the determination of all possible ordered multiplicity lists is equiv- 
alent to the solution of the IEP. Even when the two are not equivalent for some ordered list, they 
may be for all other ordered lists. 

2. In the construction of multiplicity lists for a tree, it is often useful (and, perhaps necessary) to know 
the solution of the GIEP (or some weak form of it) for some of the subtrees of the tree. 

It is often more difficult (than giving necessary restrictions) to construct matrices A € S(T) with 
a given, especially extremal, multiplicity list, even when that list does occur. There are three basic 
approaches besides ad hoc methods and computer assisted solution of equations. They are 


(a) Manipulation of polynomials, viewing the nonzero entries as variables and targeting a desired 
characteristic polynomial (see [Lea89] for an initial reference; this method, based on some nice 
formulas for the characteristic polynomial in the case of a tree (see, e.g., [Par60], [MOD89]), 
can be quite tedious for larger, more complicated trees). 


(b) Careful use of the implicit function theorem (initiated in [JSW]). 


(c) Division of the tree into understood parts and using the interlacing inequalities to give lower 
bounds that are forced to be attained by known constraints (this is along the lines of the 
brief discussion in Example 1 Section 34.2 involving A(T), but for larger trees can lead to 
complicated simultaneity conditions). 


As an example of method (c) and its subtleties (see also Example 2 Section 34.7), consider again 
the tree T in Figure 34.4. Since P(T) = 2, the maximum multiplicity is 2, and because d(T) = 3, 
there must be at least four distinct eigenvalues, two of which have multiplicity 1. This leaves the 
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question of whether the list (2,2, 1,1) (which would have to be the ordered list (1,2,2,1)) can 
occur. It can, but this is the nontrivial example with smallest number of vertices. Suppose that the 
two multiple eigenvalues are 2 and yw. We want A € S(T) with a,g(A) = 2 anda,(j) = 2. Each 
must have a Parter vertex, which must be either vertex 3 or 4. One must be for A (and not jz) and 
the other for jz (and not A), as two consecutive eigenvalues cannot share a Parter vertex (Fact 8 
section 34.1). So assume that 3 is Parter for A and 4 for jw. Then, we must have A[{1}] = 2 = A[{2}] 
and A € o(A[{4,5,6}]); and A[{5}] = uw = A[{6}] and uw € o(A[{1,2,3}]). A calculation 
(or other methods) shows this can be achieved simultaneously. 


34.6 Generalized Stars 


Definitions: 


A tree T in which there is at most one vertex of degree greater than two is a generalized star. 

In a generalized star, a vertex v is a central vertex if its neighbors are pendant vertices of their branches, 
and each branch is a path. (Note that, under this definition, a path is a (degenerate) generalized star, in 
which any vertex is a central vertex. When referring to a path as a generalized star, one vertex has been 
fixed as the central vertex.) 

For a central vertex v of a generalized star T, each branch of T at v is called an arm of T; the lengths 
of an arm are the number of vertices in the arm. 

Supposing that v is a central vertex of a generalized star T, with d;(v) = k. Denote by T,,... , Ti its 
arms and by];,... , J, the lengths of T,,... , T;, respectively. 

A star on n vertices is a tree in which there is a vertex of degree n — 1. 

Let u = (uy,...,Up), Uy) > +++ > Up, andv = (y,...,V%-), V1 > +++ = Ve, be two nonincreasing 
partitions of integers M and N, respectively. If M < N, denote by u, the partition of N obtained from u 
appending 1s to the partition u. Note that if M = N, then u, = u. 


Facts: 


1. A star is trivially a generalized star. 

2. If u and v are two nonincreasing partitions of integers M and N, respectively, M < N, such that 
uj +-:-++us <v)+---+y; for alls (interpreting u, or v, as 0 whens exceeds b or c, respectively), 
then trivially v majorizes u,, denoted u, < v. See Preliminaries. 

3. [JLSO3b] Let T be a generalized star on n vertices with central vertex v of degree k, l,... ,1; be 
the lengths of the arms T),... , Tx, and f(x), gi(x),... 5 g%(x) be monic polynomials with all their 
roots real in which deg f = n, degg; =|),... , deg gx = Ix. There exists A € S(T) such that A 
has characteristic polynomial f(x) and A[T;] has characteristic polynomial g;(x) if and only if 


* Each g;(x) has only simple roots. 


¢ IfA is a root of gi(x)---gx(x) of multiplicity m > 1, then A is a root of f(x) of multiplicity 
m—1. 

* The roots of f(x) that are not roots of g1(x)---g4(x) are simple and strictly interlace the set of 
roots of g1(x)--- g%(x) (multiple roots counting only once). 


4. [JL02b] Let T be a generalized star on n vertices with central vertex of degree s and arm lengths 
I, >--->1,.Then(p),...,p,) € £,(T) ifand only if 


© Vie1 Pi =n. 
er>]+h+i. 
* Ph = Phot =-++ = pr = lin whichh = [7"}. 


© (Pts Pas-++ > Pr—h—i) X IF 1... Uf — 1). 
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Ty To T3 


FIGURE 34.12 TT), T,, and T; are generalized stars on 9 vertices with central vertices v), v2, and v3, respectively. 


5. [JLSO3b] Let T be a generalized star on n vertices with central vertex of degree s and arm lengths 
I, >--->1,. Leta; <--- <A, be any sequence of real numbers. Then there exists a matrix A € 
S(T) with distinct eigenvalues A, < --- < A, and list of ordered multiplicities q = (q),... ,q,) if 
and only if q satisfies the following conditions: 


* a1 di = 

° Ifg; > 1,thenl <i <randq;_; =1=4)41. 

* qi tl... 5di, + Ve X (lb... 5/s)*, in which qi, = --- = qi, are the entries of the r-tuple 
(qi,.-. qr) greater than 1. 


(Thatis, when T isa generalized star, there is equivalence of the ordered multiplicity lists and the IEP. ) 


Examples: 
1. Let T,, Th, and T; be the generalized stars in Figure 34.12. We have 


£,(T) = {0 1,1,1,1,1,1,1, 1), (2,1,1,1,1,1,1, )}, 

£,(G) = {0,1,1,1,1,1,1,1, 1), (2,1, 1,1, 1, 1,1, 1), (2,2, 1,1,1,1, 1), 
(3,1, 1,1, 1,1, 1), (3,2, 1, 1,1, 1), (3,3, 1,1, 1), (4,1, 1,1,1, 1), 
(4, 2,1, 1,1)}, 


and 
L£,(T3) = {(1, 1, 1, 1, 3 ls 1, 1, 1), (2, 1, 1, 1, 1, 1, 1, 1), (2, 2, 1, 1, 1, a 1), 


(3, 1, 1, 1, 1, 1, 1), (3, 2, 1, 1, 1, 1), (3, 3, 1, 1, 1), (4, 1, 1, 1, 1, 1), 
(4, 2, 1, 1, 1), (5, 1, 1, 1, 1), (6, 1, 1, 1), (7, 1, 1)}. 


34.7. Double Generalized Stars 


Definitions: 


A double generalized star is a tree resulting from joining the central vertices of two generalized stars 
T, and T, by an edge. Such a tree will be denoted by D(T), T)). 
A double star is a double generalized star D(T,, T,) in which T, and T, are stars. 
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Sy Je hi Jg-t Sq 
FIGURE 34.13 A double path. 


A double path is a double generalized star D(T,, T,) in which T, and T) are paths. When we refer to 
a double path T on n = p + q vertices we suppose T is represented as in Figure 34.13, in which the 
only constraint on the connecting edge {i,, jy} is that not both k € {1, p} and! € {1, q}. The upper (7) 
path has k — 1 vertices to the left of the connecting vertex and another p — k vertices to the right; set 
s; = min{k — 1, p —k}, s. = min{/ — 1, q — J}, ands = min{q, p, s; +5}. 

Let G bea tree. Let v be a vertex of G of degree k and Gj,... , Gz be the components of G \ {v} having 
order J,,... 14, respectively. To the tree G is associated the generalized star, S,(G), with central vertex 
v of degree k, and with arms T),... , T; of lengths 1,,... ,J,, respectively. 

Let u; and uz be adjacent vertices of a tree G. Denote by G,,, the connected component of G\ {u2} that 
contains u, and by G,,, the connected component of G \ {u;} that contains uj. Put S; = S,,(G,,) and 
So = S,,(Gy,). Now, to the tree G is associated the double generalized star D(S), S)), which is denoted 
by Du,u,(G). 

Given a vertex v of a tree T and an eigenvalue A of a matrix A € S(T), A is an upward eigenvalue of A 
at v if gy)(A) = ag(A) + 1, and aw,4(A) is an upward multiplicity of A at v. 

If q = q(A) = (q1,...,4q,) is the list of ordered multiplicities of A, define the list of upward multi- 
plicities of A at v, denoted by q, as the list with the same entries as q but in which any upward multiplicity 
qi of A at v is marked as qj in q. 

Given a generalized star T with central vertex v, we denote by £,(T) the set of all lists of upward 
multiplicities at v occurring among matrices in S(T). 


Facts: 
1. A double path is a tree whose path cover number is 2. 
2. [JLSO3b] Let T be a generalized star on n vertices with central vertex v of degree k and arm lengths 
I >--- > q. Leta; <-+-- <A, be any sequence of real numbers. Then there exists a matrix A in 


S(T) with distinct eigenvalues A; <--- < A, anda list of upward multiplicities q = Grea 
if and only if q satisfies the following conditions: 
* a1 di = 1. 


¢ If q; is an upward multiplicity in g, then 1 <1 < r and neither qj_) nor qj—; is an upward 
multiplicity in q. 

° (qi, +1...54i, De X (lis... 5I)*, in which qj, > --- > qi, are the upward multiplicities 
of q. 


ee 


3. [JLSO3b] (Superposition Principle) Let D(T), T,) be a double generalized star, b= (b,..., b,,)€ 


£,(T;),andé = (C1)..+ 3€s) € £,(T)).Construct anybt+ = (bf,... b+, )andct = (hes ree by 


subject to the following conditions: 


* t,t €Noands; +t =s. +h. 
* b* (respectively, ct) is obtained from b (respectively, ¢) by inserting ft (respectively, ft) Os. 
* b;* and c;' cannot both be 0. 


* Ifb;* > Oandc;* > 0, then at least one of the b; or c;* must be an upward multiplicity of b or é. 


i 


) 
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Then b+ + c+ € L,(D(T, 1). Moreover, a € £L,(D(T;, T))) if and only if there are b € £,(T)), 
é € £,(D) such thata = bt + ct. 
4. Let T bea tree, v be a vertex of T, and vj, v2 be adjacent vertices of T. Then 


* Lu(S)(T)) S Lu(T), Lo(Sy(T)) € £o(T). 
* Lu(Dyiv(T)) S Lul(T), Lo(Dyv,(T)) S £o(T). 
5. [JL02b] Let T be a double path on n = p + q vertices and suppose that A € S(T). Then 


¢ The maximum multiplicity of an eigenvalue of A is 2. 


* The diameter of G is max{p, q, p+q —(s1 +52)}— 1,so that A has at least max{p, q, p+q— 
(s; + s2)} distinct eigenvalues. 


* Ahas at most s multiplicity 2 eigenvalues. 


* The possible list of unordered multiplicities for T, £,,(T), consists of all partitions of p + q into 
parts each one not greater than two and with at most s equal to 2. 


° Any list in £,(T) has at least n — 2s 1s. 


Examples: 


1. Let T; and T, be the stars in Figure 34.14 with central vertices v) and v2, respectively, and G be the 
double star D(T;, T,). By Fact 2, we have that 


£,(T) = {(,2, 1), 0, 1,1, 1), 0,1, 1,1), (11,1, )} 
and 
£(B)S10,01), 1. 1}: 


Applying the Superposition Principle (Fact 3) to the lists of upward multiplicities of T; and Th, it 
follows that 


L(G) = {(1, 3,2, 1), (1, 2, 3, 1), (1,3, 1, 1, £1, 1, 3, 1,1), 1, 1,3, 1), 
(1,252; 1, Ly(s2, 1,2,1),(, 1,2, 2,1), (1,2, 1, es 1,1), 
d, 1,2, 1s 1,1), 1, 15.25 1,1), (1, 1, 1, 1,2,1), (1, 1, 1, 1, 1, 1, 1)}. 


For example, (1, 3,2, 1) € £o(G) because b = (1,2,1) € £,(T;), € = (1, 1,1) € £,(7)), and 


(1, 3,2,1) = bt +c* = (1,2, 1,0) + (0,1, 1,1). 


Ty To D(T;, Tz) 


FIGURE 34.14 Stars and a double star. 
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i Je jg da Js 


FIGURE 34.15 A double path on 11 vertices. 


Consider the double path T on 11 vertices in Figure 34.15. Let T, be the path with vertices 
ij,... 516 and T) be the path with vertices j,,... , 75 (subgraphs of T induced by the mentioned 
vertices). Because T, and T) are generalized stars with central vertices i; and j3, respectively, from 
Fact 2 we conclude that 6 = (1,1,1,1,1,1) € £,(T)) andé = (1,1,1,1,1) € £,(1). Since, for 
example, 


(1,2, 2,2,2,1,1) = b+ +ct = (0,1, 1,1, 1,1,1) + (i, 1, 1,1,0,0), 


by the Superposition Principle, we conclude that (1,2,2,2,2,1,1) € £,(T) and, therefore, 
(2,2,2,2,1,1,1) € £,(T). We may construct a matrix A € S(T) with list of multiplicities 
(2, 2, 2, 2, 1, 1, 1) in the following way. 

Pick real numbers A; > fy > Az > M2 > Az > M3 > Ag. Construct A; with graph T, 

such that A, has eigenvalues A, (11, A2, {42, 3, A4 and such that the eigenvalues of A;[{i1, i2}] and 
Ay [{t4, is, i6}] are (41, {42 and j2), [42, 43, respectively; construct A) with graph 7; such that A) has 
eigenvalues (11, A2, 42; A3, (43 and such that the eigenvalues of both A2[{ ji, j2}] and A2[{ ja, j5}] are 
A, A3. According to Fact 3 section 34.6, these constructions are possible. Now construct A with 
graph T and such that A[T,] = A; and A[T)] = A). Then i; is a strong Parter vertex for 11, [42, 
while j3 is a strong Parter for 42,43 and so (2,2, 2, 2, 1, 1, 1) is the list of unordered multiplicities 
of A. 
Regarding Fact 4, the results for generalized stars or double generalized stars may be extended 
to a general tree T by associating with T either a generalized star or a double generalized star 
according to the given definitions. (See also [JLS03b, Theorem 10] for a corresponding result for 
the GIEP.) 

Note that, under our definition, there are many different possibilities to associate either a gener- 
alized star or a double generalized star to a given tree T, and so Fact 3 provides many possible lists 
for £(T). The natural question is to ask whether all the elements of £(T) can be obtained in this 
manner. The answer is no. It suffices to note that the path cover number of T will be, in general, 
strictly greater than that of either of S,(T) or D,,,y,(T) for any possible choice of v, v;, and v2. So 
any lists for which the maximum multiplicity occurs cannot generally be obtained from the inclu- 
sions in Fact 4. For example, the tree T in Figure 34.16 has path cover number 3, which is strictly 
greater than the maximum path cover number 2, of any generalized star or double generalized star 
associated with T. 


FIGURE 34.16 A tree with path cover number 3. 
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34.8 Vines 


Definitions: 


A binary tree is a tree in which no vertex has degree greater than 3. 
A vine is a binary tree in which every degree 3 vertex is adjacent to at least one vertex of degree 1 and 
no two vertices of degree 3 are adjacent. 


Facts: 
1. [JSW] Let T be a vine on n vertices. The set £,,(T') consists of all sequences that are majorized by 
the sequence s = (P(T),1,...,1) (s being a partition of n). 
(The description of £,,(T) was given by using the implicit function theorem technique referred to 
in section 34.5.) 
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A partial matrix is a rectangular array of numbers in which some entries are specified while others are 
free to be chosen. A completion of a partial matrix is a specific choice of values for the unspecified entries. 
A matrix completion problem asks whether a partial matrix (or family of partial matrices with a given 
pattern of specified entries) has a completion of a specific type, such as a positive definite matrix. In some 
cases, a “best” completion is sought. 

Matrix completion problems arise in applications whenever a full set of data is not available, but it 
is known that the full matrix of data must have certain properties. Such applications include molecular 
biology and chemistry (see Chapter 60), seismic reconstruction problems, mathematical programming, 
and data transmission, coding, and image enhancement problems in electrical and computer engineering. 

A matrix completion problem for a family of partial matrices with a given pattern of specified entries is 
usually studied by means of graphs or digraphs. If a pattern of specified entries does not always allow com- 
pletion to the desired type of matrix, conditions on the entries that will allow such completion are sought. 
A question of finding the “best completion” often involves optimization techniques. Matrix completion 
results are usually constructive, with the result established by giving a specific construction for a completion. 

In this chapter, we focus on completion problems involving classes of matrices that generalize the 
positive definite matrices, and emphasize graph theoretic techniques that allow completion of families of 
matrices. This chapter is organized by class of matrices, with the symmetric classes first. The authors also 
maintain a Web page containing updated information [HW]. 
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Organizing the information by classes of matrices provides easy access to results about a particular class, 
but obscures techniques that apply to the matrix completion problems of many classes and relationships 
between completion problems for different classes. For information on these subjects, see for example 
[FJT00], [Hog01], and [Hog03a]. 


35.1 Introduction 


All matrices and partial matrices discussed here are square. Graphs allow loops but do not allow multiple 
edges. The definitions and terminology about graphs and digraphs given in Chapter 28 and Chapter 29 
is used here; however, the association between matrices and digraphs is different from the association 
in those chapters, where an arc is associated with a nonzero entry. Here an arc is associated with a spe- 
cified entry. 


Definitions: 


A partial matrix is a square array in which some entries are specified and others are not. An unspecified 
entry is denoted by ? or by xj. An ordinary matrix is considered a partial matrix, as is a matrix with no 
specified entries. 

A completion of a partial matrix is a choice of values for the unspecified entries. 

A partial matrix B is combinatorially symmetric (also called positionally symmetric) if b; specified 
implies jj specified. 

Let B be ann x n partial matrix. The digraph of B, D(B) = (V, E), has vertices V = {1,...,n}, and 
for each i and j in V, the arc (i, j) € E exactly when bj is specified. 

Let B be an n x n combinatorially symmetric partial matrix. The graph of B, 9(B) = (V, E), has 
vertices V = {1,...,n}, and for each i and j in V, the edge {i, j} € E exactly when bj; is specified. 

A connected graph or digraph is nonseparable if it does not have a cut-vertex. 

A block of a graph or digraph is a maximal nonseparable sub(di)graph. This use of “block” is for graphs 
and digraphs and differs from “block” in a block matrix. 

A graph (respectively, digraph) is a clique if every vertex has a loop and for any two distinct vertices 
u,v, the edge {u, v} is present (respectively, both arcs (u,v), (v, u) are present). 

A graph or digraph is block-clique (also called 1-chordal) if every block is a clique. 

A digraph G = (V, E) is symmetric if (i, 7) € E implies (j,1) € E for alli, j € V. 

A digraph G = (V, E) is asymmetric if (i, 7) € E implies (j,i) ¢ E for all distinct i,j € V. 

A simple cycle in a digraph is an induced subdigraph that is a cycle. 

A digraph (respectively, graph) G has the X-completion property (where X is a type of matrix) if every 
partial X-matrix B such that D(B) = G (respectively, §(B) = G) can be completed to an X-matrix. In 
the literature, the phrase “has X-completion” is sometimes used for “has the X-completion property.” 

A class X is closed under permutation similarity if whenever A is an X-matrix and P is a permutation 
matrix, then P? AP is an X-matrix. 

A class X is hereditary (or closed under taking principal submatrices) if whenever A is an X-matrix 
anda C {1,...,n}, then Ala] is an X-matrix. 

A class X is closed under matrix direct sums if whenever A), Az,..., A, are X-matrices, then 
A; ® Ar ®--- @ Ax is an X-matrix. 

A class X has the triangular property if whenever A is a block triangular matrix and every diagonal 
block is an X-matrix, then A is an X matrix. 

A partial matrix B is in pattern block triangular form if the adjacency matrix of D(B) is in block 
triangular form. 

Note: Many matrix terms, such as size, entry, submatrix, etc., are applied in the obvious way to partial 
matrices. 
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Facts: 


Let X be one of the following classes of matrices: positive (semi)definite matrices, Euclidean distance ma- 
trices, (symmetric) M-matrices, (symmetric) Mo-matrices, (symmetric) inverse M-matrices, completely 
positive matrices, doubly nonnegative matrices, (strictly) copositive matrices, P-matrices, Po-matrices, 
Po,1-matrices, nonnegative P-matrices, nonnegative Po-matrices, positive P-matrices, entry (weakly) 
sign symmetric P-matrices, entry (weakly) sign symmetric Po-matrices, entry (weakly) sign symmetric 
Po,:-matrices. (The definitions of these classes can be found in the relevant sections.) 

Proofs of the facts below can be found in [Hog01] for most of the classes discussed, and the proofs given 
there apply to all the classes X listed above. Most of these facts are also in the original papers discussing the 
completion problem for a specific class; those references are listed in the section devoted to the class. In 
the literature, when it is assumed that a partial matrix B has every diagonal entry specified (equivalently, 
every vertex of the graph §(B) or digraph D(B) has a loop), it is customary to suppress all the loops and 
treat S(B) or D(B) asa simple graph or digraph. That is not done in this chapter because of the danger of 
confusion. Also, in some references, such as [Hog01], a mark is used to indicate a specified vertex instead 
of a loop. This has no effect on the results (but requires translation of the notation). If X is a symmetric 
class of matrices, then there is no loss of generality in assuming every partial matrix is combinatorially 
symmetric and this assumption is standard practice. 


1. If B isa combinatorially symmetric partial matrix, then D(B) is a symmetric digraph and 5(B) is 
the graph associated with D(B). Combinatorially symmetric partial matrices are usually studied by 
means of graphs rather than digraphs, and it is understood that the graph represents the associated 
symmetric digraph. 

2. Each of the classes X listed at the beginning of the facts is closed under permutation similarity. This 
fact is not true for the classes of totally nonnegative and totally positive matrices. (See [FJS00] for 
information about matrix completion problems for these matrices.) 

3. Applying a permutation similarity to a partial matrix B corresponds to renumbering the vertices 
of the digraph D(B) (or graph 9(B) if B is combinatorially symmetric). 

4. Renumbering the vertices of a graph or digraph does not affect whether it has the X-completion 
property. It is customary to use unlabeled (di)graph diagrams. This fact is not true for the classes 
of totally nonnegative and totally positive matrices. 

5. Each of the classes X listed at the beginning of the facts is hereditary. 

6. Let B be a partial matrix and @ C {1,...,n}. The digraph of the principal submatrix B[aq] is 
isomorphic to the subdigraph of D(B) induced by a (and is customarily identified with it). The 
same is true for the graph if B is combinatorially symmetric. 

7. Ifa graph or digraph G has the X-completion property, then every induced subgraph or induced 
subdigraph of G has the X-completion property. 

8. Each of the classes X listed at the beginning of the facts is closed under matrix direct sums. 

9. Let B beapartial matrix such that all specified entries are contained in diagonal blocks B;, By,..., Bx. 
The connected components of D(B) are isomorphic to the D(B;), i = 1,...,k. The same is true 
for §(B) if B is combinatorially symmetric. 

10. A graph or digraph G has the X-completion property if and only if every connected component 
of G has the X-completion property. 

11. If X has the triangular property, B is a partial matrix in pattern block triangular form, and each 
pattern diagonal block can be completed to an X-matrix, then B can be completed to an X- 
matrix. 

12. IfX has the triangular property and is closed under permutation similarity, then a graph or digraph 
G has the X-completion property if and only if every strongly connected component of G has the 
X-completion property. 

13. A block-clique graph is chordal. 

14. A block-clique digraph is symmetric. 
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Examples: 


1. Graphs (a) through (n) will be used in the examples in the following sections. 


(d) 


(e) 


Raves 


(g) 


Matrix Completion Problems 35-5 


ny) O . 
a, 
® 
(j) 
@ 
(k) 
(1) 
(m) 
(n) 
1 2 2 0 
Pa oe Le, ST oR) : : pee : : 
2. The matrix |, 7 12 is a partial matrix specifying the graph 1a with vertices numbered 
8 g 0 1 


1, 2, 3, 4 clockwise from upper left (or any other numbering around the cycle in order). 
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3. The graph 1f is block-clique, and this is the only block-clique graph in Example 1. 
4. The following digraphs ((a) through (r)) will be used in the examples in the following sections. 
Note that when both arcs (i, 7) and (j,7) are present, the arrows are omitted. 


(g) 
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(h) 


(j) 


ala 


35-8 Handbook of Linear Algebra 


(p) 


BO 


(r) 


5. None of the digraphs in Example 4 are symmetric. (We diagram a symmetric digraph by its 
associated graph.) Digraphs 4b, 4h, 4i, 4j, and 41 are asymmetric. 


35.2 Positive Definite and Positive Semidefinite Matrices 


In this section, all matrices are real or complex. 


Definitions: 


The matrix A is positive definite (respectively, positive semidefinite) if A is Hermitian and for all x 4 0, 
x* Ax > 0 (respectively, x* Ax > 0). 

The partial matrix B is a partial positive definite matrix (respectively, partial positive semidefinite 
matrix) if every fully specified principal submatrix of B is a positive definite matrix (respectively, positive 
semidefinite matrix), and whenever bj is specified then so is bj and ji = bij. 


Facts: 


1. A Hermitian matrix A is positive definite (respectively, positive semidefinite) if and only if it 
is positive stable (respectively, positive semistable) if and only if all principal minors are positive 
(respectively, nonnegative). There are many additional characterizations. (See Chapter 8.4 for more 
information.) 

2. [GJS84] A graph that has a loop at every vertex has the positive definite (positive semidefinite) 
completion property if and only if it is chordal. (For information on how to construct such a 
completion, see [GJS84] and [DG81].) 
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[GJS84] A graph has the positive definite completion property if and only if the subgraph induced 
by the vertices with loops has the positive definite completion property. 

[Hog01] A graph G has the positive semidefinite completion property if and only if for each 
connected component H of G, either H has a loop at every vertex and is chordal, or H has no 
loops. 

[GJS84] If B is a partial positive definite matrix with all diagonal entries specified such that $(B) is 
chordal, then there is a unique positive definite completion A of B that maximizes the determinant, 
and this completion has the property that whenever the i, j-entry of B is unspecified, the i, j-entry 
of A~! is zero. 

[Fie66] Let C be a partial positive semidefinite matrix such that every diagonal entry is specified 
and §(B) with loops suppressed is a cycle. If any diagonal entry is 0, then B can be completed to a 
positive semidefinite matrix. If every diagonal entry is nonzero, there is a positive diagonal matrix 
D such that every diagonal entry of C = DBD is equal to 1. Let the specified off-diagonal entries 
of C be denoted c;,...,¢,. Then C (and, hence, B) can be completed to a positive semidefinite 
matrix if and only if the following cycle conditions are satisfied: 


2 max arccos|c,| < Uj_,arccos|c;| for c)...Cn > 0, 
1<k<n - 


Lee arccos|cy,| > for c;...C, < 0. 


(See [BJL96] for additional information.) 


Examples: 


The graphs for the examples can be found in Example 1 of Section 35.1. 


DNs oe 


. The graphs 1d, 1f, and 1h have both the positive definite and positive semidefinite completion 


properties by Fact 2. 

The graphs 1a, 1b, 1c, le, and 1g have neither the positive definite nor the positive semidefinite 
completion property by Fact 2. 

The graphs 1j, 1k, 1, 1m, and 1n have the positive definite completion property by Facts 3 and 2. 
The graph 1i does not have the positive definite completion property by Facts 3 and 2. 

The graph 11 has the positive semidefinite completion property by Fact 4. 

The graphs 1i, 1j, 1k, 1m, and 1n do not have the positive semidefinite completion property by 
Fact 4. 


The partial matrix B = can be completed to a positive semidefinite 
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matrix by Fact 6 because 


Lp arccos|cx| = 4.10617 > z. 


35.3 Euclidean Distance Matrices 


In this section, all matrices are real. 


Definitions: 


The matrix A = [aj] is a Euclidean distance matrix if there exist vectors x),...,X, € R? (for some 
d > 1) such that aj = ||x; — xj|l2 for alli, j =1,...,n. 
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The partial matrix B is a partial Euclidean distance matrix if every diagonal entry is specified and 
equal to 0, every fully specified principal submatrix of B is a Euclidean distance matrix, and whenever bj 
is specified then so is bj; and bj; = bj. 


Facts: 


1. Every Euclidean distance matrix has all diagonal elements equal to 0. There is no loss of generality 
by considering only a graph that has loops at every vertex, and requiring all diagonal entries of 
partial Euclidean distance matrices to be 0. 

2. [Lau98] A graph with a loop at every vertex has the Euclidean distance completion property if and 
only if it is chordal. 

3. [Lau98] A graph with a loop at every vertex has the Euclidean distance completion property if and 
only if it has the positive semidefinite completion property. There is a method for transforming the 
Euclidean distance completion problem into the positive semidefinite completion problem via the 
Schoenberg transform that provides additional information about conditions on entries that are 
sufficient to guarantee completion. 


Examples: 
The graphs for the examples can be found in Example 1 of Section 35.1. 


1. The graphs 1d, 1f, and 1h have the Euclidean distance completion property by Fact 2. 
2. The graphs 1a, 1b, 1c, le, and 1g do not have the Euclidean distance completion property by Fact 2. 


35.4 Completely Positive and Doubly Nonnegative Matrices 


In this section, all matrices are real. 


Definitions: 


The matrix A is a completely positive matrix if A = CC’ for some nonnegative n x m matrix C. 
A matrix is a doubly nonnegative matrix if it is positive semidefinite and every entry is nonnegative. 
The partial matrix B is a partial completely positive matrix (respectively, partial doubly nonnegative 
matrix) if every fully specified principal submatrix of B is a completely positive matrix (respectively, 
doubly nonnegative matrix), and whenever bj is specified then so is bj; and bj; = bj, and all specified 
off-diagonal entries are nonnegative. 


Facts: 


1. A completely positive matrix is doubly nonnegative. 

2. [DJ98] A graph that has a loop at every vertex has the completely positive completion property 
(respectively, doubly nonnegative completion property) if and only if it is block-clique. 

3. [Hog02] A graph G has the completely positive completion property (respectively, doubly nonneg- 
ative completion property) ifand only if for every connected component H of G, H is block-clique, 
or H has no loops. 

4. A graph has the completely positive completion property ifand only ifit has the doubly nonnegative 
completion property. 

5. [DJK00] A partial matrix that satisfies the conditions of Fact 6 of Section 35.2 can be completed to 
a CP- (respectively, DN-) matrix. 


Examples: 
The graphs for the examples can be found in Example 1 of Section 35.1. 


1. The graph 1f has both the completely positive completion property and the doubly nonnegative 
completion property by Fact 2. 
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2. 
BE 


4. 
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The graphs 1a, 1b, 1c, 1d, le, 1g, and 1h have neither the completely positive completion property 
nor the doubly nonnegative completion property by Fact 2. 

The graph 11 has both the completely positive completion property and the doubly nonnegative 
completion property by Fact 3. 

The graphs 1i, 1j, 1k, 1m, and 1n have neither the completely positive completion property nor the 
doubly nonnegative completion property by Fact 3. 


Copositive and Strictly Copositive Matrices 


In this section, all matrices are real. 


Definitions: 


The symmetric matrix A is strictly copositive if x! Ax > 0 for allx > 0 and x + 0; A is copositive if 
x! Ax > 0 forall x > 0. 


The 


partial matrix B is a partial strictly copositive matrix (respectively, partial copositive matrix) if 


every fully specified principal submatrix of B is a strictly copositive matrix (respectively, copositive matrix) 
and whenever bj; is specified then so is bj; and bj; = bj. 


Facts: 


1. 
2. 


If A is (strictly) copositive, then so is A + M for any symmetric nonnegative matrix M. 
[HJRO5] [Hog] Every partial strictly copositive matrix can be completed to a strictly copositive 
matrix using the method described in Facts 4 and 5 below. 


. [HJRO5] Every partial copositive matrix that has every diagonal entry specified can be completed to 


a copositive matrix using the completion described in Fact 4 below. There exists a partial copositive 
matrix with an unspecified diagonal entry that cannot be completed to a copositive matrix (see 
Example 2 below). 

[HJRO5] Let B be a partial copositive matrix with every diagonal entry specified. For each pair of 
unspecified off-diagonal entries, set xj = xj: = J bid 5; . The resulting matrix is copositive, and is 
strictly copositive if B is a partial strictly copositive matrix. 

[Hog] Any completion ofa partial strictly copositive matrix omitting only one diagonal entry found 
by Algorithm 1 is a strictly copositive matrix. If B is a partial strictly copositive matrix that omits 
some diagonal entries, values for these entries can be chosen one at a time using Algorithm 1, using 
the largest value obtained by considering all principal submatrices that are completed by the choice 
of that diagonal entry, to obtain a partial strictly copositive matrix with specified diagonal that 
agrees with B on every specified entry of B. 


Algorithm 1: Completing one unspecified diagonal entry 
X11 b £ 
b B, 
except the 1,1-entry specified. Let || - || be a vector norm. 
Complete B by choosing a value for x;; as follows: 


Let B= | | be a partial strictly copositive n x n matrix having all entries 


1. B = minyeg-tysoyyjai bTY- 
2. Y = MiNycR™! y>ollyl[=1 y’ Buy. 


2 
3. x1, > ©. 
11 7 
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6. [HJR05], [Hog] Every graph has the strictly copositive completion property. 
7. [HJRO5], [Hog] A graph has the copositive completion property if and only if for each connected 
component H of G, either H has a loop at every vertex, or H has no loops. 


Examples: 
X11 —5 1 x14 X15 X16 
—5 1 -—2 X24 X25 1 
; ; 1 —2 5 SO) a . ; oe ; 
1. The partial matrix B = is a partial strictly copositive matrix. 
Xj4 X24 1 1 X45 1 
X15 X25 1 X45 X55 X56 
X16 1 -l 1 X56 3 


We use the method in Facts 4 and 5 to complete B to a strictly copositive matrix: 

Select index 5. The only principal submatrix completed by a choice of bs5 is B[{3,5}]. Any value 
that makes x55b33 > b3. will work; we choose x55 = 1. 

Select index 1. The only principal submatrices completed bya choice of b,; are principal submatrices 


X11 


bb BLI2,3}] . Apply Algorithm 1 (using || - ||1): 


of B[{1,2,3}] = | 


1. B = MiNj\y|),=1 b’y = —5. 
2yv= minjlyi),=1 ¥’ BI{2; 3} ly = T 
3. Choose x}; > E; we choose by; = 256. 


256 —5 1 16 16 1673 


—5 Ti S27 od 1 1 
1 —2 5 1 -l ro td Re : ae : 
Then by Fact 4, B = is a strictly copositive matrix. 

16 1 1 1 1 
1 1 -l 1 1 V3 

6/3 1 -1 1 V3 3 

St Ie : ae : re : 
2,B= Ee ‘ is a partial copositive matrix that cannot be completed to a copositive matrix 


because once x, is chosen (clearly x;; > 0), then v = [ee 1]? results inv? By = = 


3. The graphs 1a, 1b, 1c, 1d, le, 1f, 1g, 1h, and 1] have copositive completion by Fact 7, and these are 
the only graphs in Example 1 of section 35.1 that have the copositive completion property. 


<0. 


35.6 M- and M,-Matrices 


In this section, all matrices are real. 


Definitions: 


The matrix A is an M-matrix (respectively, Mo-matrix) if there exist a nonnegative matrix P and a real 
number s > p(P) (respectively, s > p(P)) such that A = sI — P. 

The partial matrix B is a partial M-matrix (respectively, partial Mo-matrix) if every fully specified 
principal submatrix of B is an M-matrix (respectively, Mo-matrix) and every specified off-diagonal entry 
of B is nonpositive. 

If B is a partial matrix that includes all diagonal entries, the zero completion of B, denoted Bo, is 
obtained by setting all unspecified (off-diagonal) entries to 0. 
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Facts: 
1. Fora Z-matrix A (i.e., every off-diagonal entry of A is nonpositive), the following are equivalent: 


(a) Ais an M-matrix. 

(b) Every principal minor of A is positive. 
(c) A is positive stable. 

(d) A7! is nonnegative. 


The analogs of the first three conditions are equivalent to A being an Mo-matrix. See Section 9.5 
for more information about M- and Mo-matrices. 

2. A principal submatrix of an M- (Mp-) matrix is an M- (Mo-) matrix (cf. Fact 5 in Section 35.1). 

3. [JS96], [Hog01] A partial M- (Mo-) matrix B that includes all diagonal entries can be completed 

to an M- (M-) matrix if and only if the zero completion Bo is an M- (Mp-) matrix. 

4. [Hog98b], [Hog01] A digraph G with a loop at every vertex has the M- (Mp-) completion property 

if and only if every strongly connected induced subdigraph of G is a clique. 

5. [Hog98b] A digraph G has the M-completion property if and only if the subdigraph induced by 

the vertices of G that have loops has M-completion. 

6. [Hog01] A digraph G has the Mo-completion property if and only if for every strongly connected 

induced subdigraph H of G, either H is a clique or H has no loops. 

7. [Hog02] Symmetric M- and Mo-matrices and partial matrices are defined in the obvious way. A 
graph has the symmetric M-completion property if and only if every connected component of the 
subgraph induced by the vertices with loops is a clique. A graph has the symmetric My-completion 
property if and only if every connected component is either a clique or has no loops. 


Examples: 
The graphs and digraphs for the examples can be found in Examples 1 and 4 of Section 35.1. 


1. Even if the digraph of a partial M-matrix does not have the M-matrix completion property, the 
matrix may still have an M-matrix completion. By Fact 3 it is easy to determine whether there 


1 ¢ C72 
—05 2 # 2 
is an M-completion. For example, complete the partial M-matrix B = ; i ; ; 

? eS] 1 


to Bo by setting every unspecified entry to 0. To determine whether Bo is an M-matrix, com- 
pute the eigenvalues: o( By) = {2.38028, 1.21945 + 0.914474i, 0.180827}. If we change the val- 
ues of entries we can obtain a partial M-matrix that does not have an M-matrix completion, 


1 g oS). 
a a 
e.g, C = ee ; ai Then 0 (Co) = {2.82397, 1.23609 + 1.434997, —0.296153}, so 
¢ Gust 1 
by Fact 3, C cannot be completed to an M-matrix. Note D(B) = D(C) is the digraph 4i, which 
does not have the M-matrix completion property by Fact 4. 
2. The digraphs 4j, 4m, 4p, and 4q have the M- and Mo-completion properties by Fact 4. 
3. The graphs 1a, 1b, 1c, 1d, le, 1f, 1g, and 1h and the digraphs 4f, 4g, 4h, 4i, 4k, 41, 4n, 40, and 4r 
have neither the M-completion property nor the Mo-completion property by Fact 4. 
4. The graphs 1j, 11, and 1m and the digraphs 4a, 4b, 4c, and 4d have the M-completion property by 
Facts 5 and 4. Of these (di)graphs, only 1] and 4c have the My completion property, by Fact 6. 
5. The graphs 1i, 1k, and 1n and the digraph 4e do not have the M-completion property (respectively, 
the Mo-completion property) by Facts 5 and 4 (respectively, Fact 6). 
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35.7. Inverse M-Matrices 


In this section, all matrices are real. 


Definitions: 
The matrix A is an inverse M-matrix if A is the inverse of an M-matrix. 
The partial matrix B is a partial inverse M-matrix if every fully specified principal submatrix of B is 
an inverse M-matrix and every specified entry of B is nonnegative. 
A digraph is cycle-clique if the induced subdigraph of every cycle is a clique. 
An alternate path to a single arc in a digraph G is a path of length greater than 1 between vertices i and 
j such that the arc (i, 7) is in G. 
A digraph G is path-clique if the induced subdigraph of every alternate path to a single arc is a clique. 
A digraph is homogeneous if it is either symmetric or asymmetric. 


Facts: 


1. A matrix A is an inverse M-matrix if and only if all entries of A are nonnegative and all off-diagonal 
entries of A~! are nonpositive. There are many equivalent characterizations of inverse M-matrices; 
see Section 9.5 for more information. 


By by ? 
2. [JS96] Let B = bj, ba bj, be an n x n partial inverse M matrix, where B,,; and B33 are 
2 bsz—- B33 


square matrices of size k and n — k — 1, and all entries of the submatrices shown are specified. Then 
By by byb3,'bh, 
A= bj, boo bi, is the unique inverse M-completion of B such that A~! has 
bs2b37'bj, 32 B33 
zeros in all the positions where B has unspecified entries. This method can be used to complete a 
partial inverse M-matrix whose digraph is block-clique. 

3. [JS96], [JS99], [Hog98a], [Hog00], [Hog02] A symmetric digraph with a loop at every vertex has 
the inverse M-completion property if and only if it is block-clique. A digraph obtained from a 
block-clique digraph by deleting loops from vertices not contained in any block of order greater 
than 2 also has the inverse M-completion property, and any symmetric digraph that has the inverse 
M-completion property has that form. The same is true for the symmetric inverse M-completion 
property (with the obvious definition). 

4. [Hog98a] A digraph with a loop at every vertex has the inverse M-completion property if and only 
if G is path-clique and cycle-clique. 

5. [Hog00], [Hog01] A digraph G has the inverse M-completion property ifand only ifit is path-clique 
and every strongly connected nonseparable induced subdigraph has the inverse M-completion 
property. A strongly connected nonseparable digraph is homogeneous. A simple cycle with at least 
one vertex that does not have a loop has the inverse M-completion property. 


Examples: 
The graphs and digraphs for the examples can be found in Examples 1 and 4 of Section 35.1. 
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2. By Fact 3, the graphs 1f and 1k have the inverse M-completion property, and these are the only 
graphs in Example 1 that do. 

3. The digraphs 4j and 4m have the inverse M-completion property by Fact 4. 

4. The digraphs 4f, 4g, 4h, 4i, 4k, 41, 4n, 40, 4p, 4q, and 4r do not have the inverse M-completion 
property by Fact 4. 

5. The digraphs 4a, 4b, 4c, 4d, and 4e do not have the inverse M-completion property by Fact 5. 


35.8 P-, Po-, and Po-Matrices 


In this section, all matrices are real. 


Definitions: 


The matrix A is a P-matrix (respectively, Po-matrix, Po )-matrix) if every principal minor of A is positive 
(respectively, nonnegative, nonnegative and every diagonal element is positive). 

The partial matrix B is a partial P -matrix (respectively, partial Po-matrix, partial Po |-matrix) if every 
fully specified principal submatrix of B is a P-matrix (respectively, Po-matrix, Po,-matrix). 


Facts: 


1. A positive definite matrix, M-matrix, or inverse M-matrix is a P-matrix. A positive semidefinite 
matrix or My-matrix is a Po-matrix. See [HJ91] for more information on P-, Po,;-, and Po-matrices. 
A principal submatrix ofa P - (Po,1-, Po-) matrixisa P - (Po,1-, Po-) matrix (cf. Fact 5 in section 35.1). 

2. [Hog03a] Ifa digraph has the Po-completion property, then it has the Po,;-completion property. If 
a digraph has the Po,;-completion property, then it has the P-completion property. 

3. [JK96], [Hog01] A digraph has the P-completion property if and only if the subdigraph induced 
by the vertices that have loops has the P-completion property. 

4. [JK96] Every symmetric digraph has the P-completion property. The P-completion of a combi- 
natorially symmetric partial P-matrix can be accomplished by selecting one pair of unspecified 
entries at a time and choosing the entries of opposite sign and large enough magnitude to make 
the determinants of all principal matrices completed positive. 

5. [JK96] Every order 3 digraph has the P-completion property, but there is an order 4 digraph 
(see the digraph 4k in Example 4 of Section 35.1) that does not have the P-completion property. 
[DH00] extended a revised version of this example, digraph 4r, to a family of digraphs, called 
minimally chordal symmetric Hamiltonian, that do not have the P -completion property. The digraph 
4n is another example of a digraph in this family (so both 4r and 4n do not have the P-completion 
property). 

6. [DH00] A digraph that can be made symmetric by adding arcs one at a time so that at each stage 
at most one order 3 induced subdigraph (and no larger) becomes a clique, has the P-completion 


property. 

7. [CDH02] A partial P- (Po1-, Po-) matrix whose digraph is asymmetric can be completed to a 
P- (Po-, Po-) matrix as follows: Ifi # j and bj; is specified, then set xj; = —bj. Otherwise, set 
xij = 0. Every asymmetric digraph has the P- (Po,:-, Po-) completion property. 

By bo 2? 
8. [FJTOO] Let B= |b}, by bi,| beann x n partial P- (Po-) matrix, where By, and B33 are 
2 bsz_—- B33 


square matrices of size k and n — k — 1, and all entries of the submatrices shown are specified. 
By by by2by'bF, 
Then A= |b by bi, is a P- (Po.-) completion of B. This method can be used to 
0 bsz B33 
complete any partial P- (Po,-) matrix whose digraph is block-clique. (See [Hog01] for the details 
of the analogous completion for Po-matrices.) 
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9. [FJT00] Every block-clique digraph has the P- (Po,1-, Po-) completion property. 
10. [Hog01] A digraph G has the P- (respectively, Po,,-, Po-) completion property if and only if every 
strongly connected and nonseparable induced subdigraph of G has the P- (respectively, Po,1-, Po-) 
completion property. A method to obtain such a completion is given in Algorithm 2. 


Algorithm 2: 
Let B bea partial P- (Po-, Po-) matrix such that every strongly connected and nonseparable 
induced subdigraph K of D(B) has the P- (Po1-, Po-) property. 


1. For each such K, complete the principal submatrix of B corresponding to K to obtain a 
partial P- (Po-, Po-) matrix B, such that each strongly connected induced subdigraph 
S of D(B;) is block-clique. 

2. For each such S, complete the principal submatrix of B, corresponding to S to obtain 
a partial P- (Po1-, Po-) matrix B. 

3. Set any remaining unspecified entries to 0. 


11. [Hog01] A digraph that omits all loops has the P- ( Po,,-, Po-) completion property. Each connected 
component of a symmetric digraph that has the Po-completion property must have a loop at every 
vertex or omit all loops. 

12. [Hog01], [CDH02], [JK96] A symmetric n-cycle with a loop at every vertex has the Po-completion 
property if and only if n # 4. A symmetric n-cycle with a loop at every vertex has the P- and 
Po,-completion properties for all n. 

13. [CDH02] All order 2, order 3, and order 4 digraphs with a loop at every vertex have been classified 
as having or not having the Po-completion property. There are some order 4 digraphs that (in 2005) 
have not been classified as to the P- and Po,-completion properties. 


Examples: 
The graphs and digraphs for the examples can be found in Examples 1 and 4 of Section 35.1. 
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1. It is easy to verify that B = is a partial P-matrix. The graph 1d, called the 


double triangle, is interpreted as the digraph of B. Let x24 = y and x4. = —y.Achoice of y completes 
three principal minors, det B[{2, 4}] = 6+y", det B[{1,2,4}] = —l-y+y’, and det B[{2, 3, 4}] = 
1l+4y+ y. The choice y = 2 makes all three minors positive. Let x24 = z and x4. = —z. With 
y = 2,a choice of z completes four principal minors, det B[{1,3}] = 1+ z’, det B[{1, 2, 3}] = 
5+6z+ 32’, det B[{1,3,4}] = 22’, and det B = 10+ 18z+ 102’, so setting z = 0 completes B 


1 2 0 -!l 
: 1 3. 2 : ; ieee ; 
to the P-matrix 0 2 1 il: Any partial P-matrix specifying the double triangle can 


a2. 1 2 


be completed in a similar manner, so the double triangle has the P -completion property. 


I 2s 2 

; . -1 0 0 —2 

2. The double triangle 1d does not have the Po-completion property because B = i. a0 i 
toad J 1 


cannot be completed to a Po-matrix ([JK96]). This implies the graphs 1g and 1h do not have the 
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Po-completion property by Fact 7 in Section 35.1. The double triangle does have the Po,;-completion 
property [Wan05]. 

3. All graphs in Example 1 have the P-completion property by Fact 4. 

4. The digraphs 4c and 4d have the P-completion property by Fact 3. Fact 3 can also be applied in 
conjunction with other facts to several other digraphs in Example 4. 

5. The digraphs 4f, 4g, 4h, 4p, and 4q have the P-completion property by Fact 5. 

6. The digraphs 4a, 4b, 4c, 4d, 4i, 4j, 41, 4m, and 40 have the P-completion property by Fact 6. 

7. The digraphs 4b, 4h, 4i, 4j, and 41 have the P-, Po,-, and Po-completion properties by Fact 7. 


1 —-l ¢ 1 —-l 2 —1 

; 3 2 —4 : . 13 2 —4 2 

8. The completion of | , 1 5 given by Fact 8 is 0 1 5 -1 
2 1 2 2 0 1 2 2 


9. The graph 1f has the Po- and Po,;-completion properties by Fact 9. (It also has the P-completion 

property but one would not normally cite Fact 9 for that.) 

10. The digraphs 4m, 4p, and 4q have the P-, Po,-, and Po-completion properties by Fact 10. Fact 10 
can also be applied in conjunction with other facts to several other digraphs in Example 4. 

11. The graph 1] and the digraph 4c have the P-, Po-, and Po,1-completion properties by Fact 11. 

12. The graphs 1i, 1j, 1k, 1m, and 1n do not have the Po-completion property by Fact 11. 

13. The graphs 1b and Ic have the Po- (Po, )-completion property by Fact 12. 

14. The graph 1a does not have the Pp-completion property, but does have the Pp ;-completion property, 
by Fact 12. 

15. The graphs leand 1gdonothave the Po-completion property by Fact 12 and by Fact 7 in Section 35.1. 


35.9 Positive P-, Nonnegative P-, Nonnegative Pp,-, 
and Nonnegative Py-Matrices 


In this section, all matrices are real. 


Definitions: 


The matrix A is a positive (respectively, nonnegative) P-matrix if A is a P-matrix and every entry of A is 
positive (respectively, nonnegative). The matrix A is a nonnegative Po-matrix (respectively, nonnegative 
Po,:-matrix) if A is a Po-matrix (respectively, Po,;-matrix) and every entry of A is nonnegative. 

The partial matrix B is a partial positive P -matrix (respectively, partial nonnegative P -matrix, partial 
nonnegative Po-matrix, partial nonnegative Po,-matrix) if and only if every fully specified principal 
submatrix of B is a positive P -matrix (respectively, nonnegative P -matrix, nonnegative Po-matrix, partial 
nonnegative Po,-matrix) and all specified entries are positive (respectively, nonnegative, nonnegative, 
nonnegative). 


Facts: 


1. [Hog03a], [Hog03b] If a digraph has the nonnegative Py-completion property, then it has the 
nonnegative Po,-completion property. Ifa digraph has the nonnegative Pp, -completion property, 
then it has the nonnegative P-completion property. Ifa digraph has the nonnegative P -completion 
property, then it has the positive P-completion property. 

2. [Hog01] A digraph has the positive (respectively, nonnegative) P-completion property if and only 
if the subdigraph induced by vertices that have loops has the positive (respectively, nonnegative) 
P-completion property. 
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[FJTO0], [Hog01], [CDH03] All order 2 and order 3 digraphs that have a loop at every vertex have 
the positive P- (nonnegative P-, nonnegative Po-, nonnegative Po,-) completion property. 
[BEH06] Suppose G is a digraph such that by adding arcs one at a time so that at each stage at most 
one order 3 induced subdigraph (and no larger) becomes a clique, it is possible to obtain a digraph 
G’ that has the positive P- (respectively, nonnegative P-, nonnegative Py-, nonnegative Po-) 
completion property. Then G has the positive P - (respectively, nonnegative P-, nonnegative Po-, 
nonnegative Po,-) completion property. 

[FJTOO] A block-clique digraph has the positive P- (nonnegative P-, nonnegative Po-, nonnegative 
Po,-) completion property. See Fact 8 of section 35.8 for information on the construction. 
[Hog01] A digraph G has the positive P- (respectively, nonnegative P-, nonnegative Po-, non- 
negative Po-) completion property if and only if every strongly connected and nonseparable 
induced subdigraph of G has the positive P- (respectively, nonnegative P-, nonnegative Po-, 
nonnegative Po,-) completion property. See Algorithm 2 of section 35.8 for information on the 
construction. 

[Hog01] A digraph that omits all loops has the positive P- (nonnegative P-, nonnegative Po,-, 
nonnegative Po-) completion property. Each connected component ofa symmetric digraph that has 
the nonnegative Po-completion property must have a loop at every vertex or omit all 
loops. 

[CDH03] An order 4 digraph with a loop at every vertex has the nonnegative Po-completion 
property ifand only ifit does not contain a 4-cycle or is the clique on 4-vertices. This characterization 
does not extend to higher order digraphs. 

[BEH06], [JTU03] All order 4 digraphs that have a loop at every vertex have been classified as to 
the positive P- (nonnegative P-) completion property. 

[CDH03], [FJT00] A symmetric n-cycle that has a loop at every vertex has the nonnegative Po- 
completion property if and only ifn # 4. A symmetric n-cycle that has a loop at every vertex has 
the positive P- (nonnegative P-, nonnegative Po,-) completion property. 

[BEH06] A minimally chordal symmetric Hamiltonian digraph (cf. Fact 5 in Section 35.8) has 
neither the positive nor the nonnegative P-completion property. 


Examples: 


The graphs and digraphs for the examples can be found in Examples 1 and 4 of Section 35.1. 


I’. 


The digraphs 4f, 4g, 4h, 4p, and 4q have the positive P- (nonnegative P-, nonnegative Pp-, non- 
negative Pp -) completion property by Fact 3. 

The graph 1fhas the positive P- (nonnegative P-, nonnegative Po-, nonnegative Po,-) completion 
property by Fact 5. 


. The graphs 1j, 1k, 11, 1m, and 1n and the digraphs 4c and 4d have the positive P- (nonnegative 


P-) completion property by Facts 2 and 5. 

The digraphs 4j, 4m, 4p, and 4q have the positive P - (nonnegative P -, nonnegative Py-, nonnegative 
Po,-) completion property by Fact 6. 

The graph 1] and the digraph 4c have the positive P-, nonnegative P-, nonnegative Po-, and 
nonnegative Po,;-completion properties by Fact 7. 

The graphs 1i, 1j, 1k, 1m, and 1n do not have the nonnegative Po-completion property by Fact 7. 
The graphs 1a and 1d and the digraphs 4i, 4k, and 4r do not have the nonnegative Po-completion 
property by Fact 8. The graphs le, 1g, and 1h and the digraphs 41, 4n, and 40 do not have the 
nonnegative Po-completion property by Fact 8, using Fact 7 of Section 35.1. 

By Fact 10, the graph 1a does not have and the graphs 1b and 1c do have the nonnegative Ppo- 
completion property. 

The graphs 1a, 1b, and 1c have the positive P- (nonnegative P-, nonnegative Po,-) completion 
property by Fact 10. 
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35.10 Entry Sign Symmetric P-, Entry Sign Symmetric Pp-, 
Entry Sign Symmetric Po,-, Entry Weakly Sign Symmetric 
P-, and Entry Weakly Sign Symmetric Po-Matrices 


In this section, all matrices are real. In the literature, entry sign symmetric is often called sign symmetric; as 
defined in Chapter 19.2, the latter term is used for a different condition. 


Definitions: 


The matrix A is an entry sign symmetric P- (respectively, Po,)-, Po-) matrix if and only if A is a P -matrix 
(respectively, Po,;-matrix, Pp-matrix) and for all i, j, either ajajj > 0 or ay = aj; = 0. 

The matrix A is an entry weakly sign symmetric P- (respectively, Po,-, Po-) matrix if and only if A is 
a P-matrix (respectively, Po,.-matrix, Po-matrix) and for all i, j, ajaji > 0. 

The partial matrix B is a partial entry sign symmetric P- (respectively, Po,-, Po-) matrix if and only 
if every fully specified principal submatrix of B is an entry sign symmetric P- (respectively, Po1-, Po-) 
matrix and if both bj and bj; are specified then bjjbj; > 0 or bj = bj = 0. 

The partial matrix B is a partial entry weakly sign symmetric P - (respectively, Po,)-, Po-) matrix ifand 
only if every fully specified principal submatrix of B is an entry weakly sign symmetric P- (respectively, 
Po,-, Po-) matrix and if both bj and bj; are specified then bjjbj; = 0. 


Facts: 


1. [Hog03a] Any pattern that has the entry sign symmetric Po- (respectively, entry weakly sign sym- 
metric Po-) completion property also has the entry sign symmetric Po, - (respectively, entry weakly 
sign symmetric Po,-) completion property. Any pattern that has the entry sign symmetric Pp,1- 
(respectively, entry weakly sign symmetric Po,-) completion property also has the entry sign sym- 
metric P- (respectively, entry weakly sign symmetric P-) completion property. 

2. [Hog01] A digraph G has the (weakly) entry sign symmetric P-completion property if and only 
if the subdigraph of G induced by vertices that have loops has the entry (weakly) sign symmetric 
P-completion property. 

3. [FJT00], [Hog01] A digraph G has the entry sign symmetric Py-completion property if and only 
if for every connected component H of G, either H omits all loops or H has a loop at every vertex 
and is block-clique. 

4, [FJT00] A symmetric digraph with a loop at every vertex has the entry sign symmetric Pp,1- 
completion property if and only if every connected component is block-clique. 

5. [FJT00] A block-clique digraph has the entry sign symmetric P- (entry sign symmetric Po-, entry 
sign symmetric Po-, entry weakly sign symmetric P-, entry weakly sign symmetric Po-, entry 
weakly sign symmetric Po-) completion property. (See Fact 8 of Section 35.8 for information on 
the construction.) 

6. [Hog01] A digraph G has the entry sign symmetric P- (entry weakly sign symmetric P-, entry 
weakly sign symmetric Po,-, entry weakly sign symmetric Pp-) completion property if and only 
if every strongly connected and nonseparable induced subdigraph of G has the entry sign sym- 
metric P- (entry weakly sign symmetric P-, entry weakly sign symmetric Pp,-, entry weakly sign 
symmetric Po-) completion property. (See Algorithm 2 of Section 35.8 for information on the 
construction.) 

7. Fact 6 is not true for the entry sign symmetric Po-matrices (cf. Fact 3) or for the entry sign 
symmetric Pp j-matrices [Wan05]. In particular, the digraphs 4p and 4q in Example 4 of 
Section 35.1 have neither the entry sign symmetric Po-completion property nor the entry sign 
symmetric Po -completion property. 

8. [DHH03] A symmetric n-cycle with a loop at every vertex has the entry sign symmetric P- (entry 
weakly sign symmetric P-, entry weakly sign symmetric Po,)-, entry weakly sign symmetric Po-) 
completion property if and only ifn 4 4andn $5. 
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9. [DHH03] An order 3 digraph G with a loop at every vertex has the entry sign symmetric P - (entry 
weakly sign symmetric P-, entry weakly sign symmetric Po,)-, entry weakly sign symmetric Po-) 
completion property if and only if its digraph does not contain a 3-cycle or is a clique. 

10. [DHH03], [Wan05] All order 4 digraphs that have a loop at every vertex have been classified as to the 
entry sign symmetric P- (entry sign symmetric Po, -, entry sign symmetric Po-, entry weakly sign 
symmetric P-, entry weakly sign symmetric Po -, entry weakly sign symmetric Po-) completion 
property. 


Examples: 
The graphs and digraphs for the examples can be found in Examples 1 and 4 of section 35.1. 


1. By Fact 3, the graphs 1f and 1] and the digraph 4c have the entry sign symmetric Po-completion 
property and none of the other graphs or digraphs pictured do. 

2. The graphs la, 1b, 1c, 1d, le, 1g, and 1h do not have the entry sign symmetric Po,,-completion 
proper by Fact 4. 

3. The graph If has the entry sign symmetric P- (entry sign symmetric Po,1-, entry sign symmetric 
Po-, entry weakly sign symmetric P-, entry sign symmetric Po, -, entry weakly sign symmetric Po-) 
completion proper by Fact 5. 

4. The graphs 1j, 1k, 1, 1m, and In and the digraphs 4c and 4d and have the entry (weakly) sign 
symmetric P-completion property by Facts 2 and 5. 

5. The digraphs 4j, 4m, 4p, and 4q have the entry sign symmetric P- (entry weakly sign symmetric 
P-, entry weakly sign symmetric Po-, entry weakly sign symmetric Po-) completion property by 
Fact 6. 

6. The graphs 1a and 1b do not have the entry sign symmetric P- (entry weakly sign symmetric P-, 
entry weakly sign symmetric Po, -, entry weakly sign symmetric Po-) completion property by Fact 
8. 

7. The graph Ic has the entry sign symmetric P- (entry weakly sign symmetric P-, entry weakly sign 
symmetric Pp )-, entry weakly sign symmetric Po-) completion property by Fact 8. 

8. The digraphs 4f, 4g, 4h, 4k, 41, 4n, 40, and 4r do not have the entry sign symmetric P- (entry 
weakly sign symmetric P-, entry weakly sign symmetric Po,)-, entry weakly sign symmetric Po-) 
completion property by Fact 9 and Fact 7 in section 35.1. 
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36.1 Algebraic Connectivity for Simple Graphs: Basic Theory 


Let G bea simple graph on n > 2 vertices with Laplacian matrix Lg, and label the eigenvalues of Lg as 
0 = py < M2 <... < fy. Throughout this chapter, we consider only Laplacian matrices for graphs on at 
least two vertices. Henceforth, we use the term graph to refer to a simple graph. 


Definitions: 


The algebraic connectivity of G, denoted a(G), is given by a(G) = po. 

A Fiedler vector is an eigenvector of Lg corresponding to a(G). 

Given graphs G; = (Vj, E;) and G2 = (V, E2), their product, G; x Gp, is the graph with vertex set 
V, x V2, with vertices (u), uz) and (w), w2) adjacent if and only if either u, is adjacent to w; in G; and 
U2 = Wy), OF U2 is adjacent to w2 in G2 and u, = wy. 


Facts: 


1. [Fie89] Let G be a graph of order n with Laplacian matrix Lg. Then a(G) = min{) 7; < j4;,j)cr 

eH) | ase a =H Be HO mn eal ee a SO, 

2. [Fie73] The algebraic connectivity of a graph is nonnegative, and is equal to 0 if and only if the 

graph is disconnected. 

3. [Fie73] Let G be a connected graph on n vertices with vertex connectivity «,(G), and suppose that 

G # K,. Then a(G) < x,(G). (See Fact 13 below for a discussion of the equality case.) 

4. [Fie73] Suppose that G is a graph on n vertices, and let G denote the complement of G. Then 
a(G) =n — [y, where j2,, denotes the largest Laplacian eigenvalue for G. 
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([GRO1], p. 280) If G is a graph on n > 2 vertices that is regular of degree k, then denoting the 
eigenvalues of Ag by A, <... < An, we have a(G) = k — Ay-1. 

[Mer94] Suppose that G; and G; are two graphs on n, and ny vertices, respectively, with n,,n2 > 2. 
Then a(G,; + G2) = min{a(G,) + m,a(G2) + nj}. Similarly, if H is a graph on k > 2 vertices, 
thena(H + K,)=a(H)+1. 

[Fie73] Suppose that G, and G) are graphs, each of which has at least two vertices. Then a(G, x 
Gy) = min{a(G_), a(G2)}. ; 

[Fie73] Suppose that the graph G is formed from the graph G by adding an edge not already present 
in G. Then a(G) < a(G). 

[FK98] Let G and H be graphs, and suppose that the graph G is formed from G U H as follows: Fix 
a vertex v of G and a subset S of the vertex set for H, and for each w € S, add in the edge between 
vy and w. Then a(G) < a(G). 

Fie73] Let G be a graph, and suppose that the graph G is formed from G by deleting a collection 
of k vertices and all edges incident with them. Then a(G) > a(G)—k. 

GMS90] Let G be a graph on n > 3 vertices and suppose that the edge e of G is not on any 
3-cycles. Form G from G by deleting e and identifying the two vertices incident with it. Then 
a(G) > a(G). 


12. [Fie73] If G is a graph on n vertices, then a(G) < n. Equality holds if and only if 
G= K,. 

13. [KMNSO2] Let G be a connected graph on n vertices with vertex connectivity «,(G), and suppose 
that G # K,,. Then a(G) = x,(G) if and only if G can be written as G = G; + G», where G, is 
disconnected, G» has x,(G) vertices, and a(G2) > 2x,(G) — n. 

14. [Fie89] If G isaconnected graph on n vertices with edge connectivity x, (G), then 2(1— cos(*))Ke(G) 
< a(G). Equality holds if and only if G = Py. 

Examples: 
1. ([GK69], p. 138) For n > 2, the algebraic connectivity of the path P,, is 2(1 — cos(%)), and it isa 


simple eigenvalue of the corresponding Laplacian matrix. 

The following can be deduced from basic results on circulant matrices. If n > 3, the algebraic 
connectivity of the cycle C,, is 2(1 — cos(=*)), and it is an eigenvalue of multiplicity 2 of the 
corresponding Laplacian matrix. 

The algebraic connectivity of K, is n, and it is an eigenvalue of multiplicity n — 1 of the corres- 
ponding Laplacian matrix. 

Ifm < nand2 <n, thena(K,,,) = m.If1 < m <n, then misan eigenvalue of multiplicity n — 1 
of the corresponding Laplacian matrix, while if 2 < m = n, then m is an eigenvalue of multiplicity 
2m — 2 of the corresponding Laplacian matrix. 

The algebraic connectivity of the Petersen graph is 2, and it is an eigenvalue of multiplicity 5 of the 
corresponding Laplacian matrix. 


. The algebraic connectivity of the ladder on 6 vertices, shown in 


Figure 36.1, is 1. 


. Graphs with large algebraic connectivity arise in the study of the 


class of so-called expander graphs. (See Section 28.5 or [Alo86] 
for definitions and discussion.) 


. A Fiedler vector for a graph provides a heuristic for partitioning 


its vertex set so that the number of edges between the two parts 
is small (see [Moh92] for a discussion), and that heuristic has 
applications to sparse matrix computations (see [PSL90]). 


FIGURE 36.1 
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36.2 Algebraic Connectivity for Simple Graphs: Further Results 


The term graph means simple graph in this section. 


Definitions: 


Let G = (V, E) bea graph, and let X, V\ X be a nontrivial partitioning of V. The corresponding edge cut 
is the set E x of edges of G that have one end point in X and the other end point in V\ X. 

Suppose that G; and G) are graphs. A graph H is formed by appending G» at vertex v of G, if H is 
constructed from G; U G» by adding an edge between v and a vertex of G. 

Suppose that g,n € Nwithn > g > 3. Thegraph C,,, is formed by appending the cycle C, at a pendent 
vertex of the path P,_¢. 

Suppose that g,n € N withn > g > 3. Let Dy, -, denote the graph formed from the cycle C, by 
appending n — g isolated vertices at a single vertex of that cycle. 

For a connected graph G, a vertex v is a cut-vertex if G — v, the graph formed from G by deleting the 
vertex v and all edges incident with it, is disconnected. 

A graph is unicyclic if it contains precisely one cycle. 


Facts: 


1. [Moh91] If G is a connected graph on n vertices with diameter d, then a(G) > =. 


2. [AM85] If G is a connected graph on n vertices with diameter d and maximum degree A, then 
d< 20 y/ seq loganl. 

3. [Moh91] If G is a connected graph on n vertices with diameter d and maximum degree A, then 
d< 2p See In(n —1)]. 

4. [Moh92] Let G = (V, E) bea graph, let X, V\X bea nontrivial partitioning of its vertex set, and let 


E x denote the corresponding edge cut. Then a(G) < PaLaba and ([FKP03a]) ifa(G) = Dalaba ; 


then necessarily there are integers d), dy such that: 
* Each vertex in X is adjacent to precisely d, vertices in V\ X. 
* Each vertex in V\ X is adjacent to precisely d) vertices in X. 
* |X|d, = |V\ X]dp. 

* a(G) = d, + do. 
5. [Moh89] Let G = (V, E) beagraph on atleast four vertices, with maximum degree A. Then 
®(G) < J/a(G)(2A — a(G)), where ®(G) is the isoperimetric number of G. (See Section 32. 5.) 
6. [FK98] Amongall connected graphs on n vertices with girth 3, the algebraic connectivity is uniquely 
minimized by C,,3. 
7. [FKP02] Ifg > 4and n > 3g — 1, then among all connected graphs on n vertices with girth g, the 
algebraic connectivity is uniquely minimized by C,,¢. 
8. [Kir00] Let G be a graph on n vertices, and suppose that G has k cut-vertices, where 2 < k < oe 


a < 


Then a(G) < ——2@-)____. For each such k and n, there is a graph on n vertices with k 
(G) s n—k+2+4/(n—k)?+4 grap 


cut-vertices such that equality is attained in the upper bound on a. 
9. [Kir01] fae ares that n > 5 and 5 < k, and let G bea graph on n vertices with k cut-vertices. Let 
q=\|+ zi andl =k —(n—k)q. 


° If] = 1, thena(G) < 2(1 — cos( 5755)). 


* If] = 0, let 4 be the unique element of Es su] such that (n — k — 1)cos((2q + 1)@9/2) + 
cos((2q + 3)6)/2) = 0. Then a(G) < 2(1 — cos(O)). 


36-4 


10. 


11. 


12. 


13. 


Handbook of Linear Algebra 


FIGURE 36.2 The graph C,3. 


* If2 <I, let 0 be the unique element of EE is such that (n — k — 1)cos((2q + 3)@9/2) + 


cos((2q + 5)6)/2) = 0. Then a(G) < 2(1 — cos(6)). 
For each k and n with 5 < k, there is a graph on n vertices with k cut-vertices such that equality 
is attained in the corresponding upper bound on a. 
[FK98] Over the class of unicyclic graphs on n vertices having girth 3, the algebraic connectivity is 
uniquely maximized by D3,,_3. 
[FKP03b] Over the class of unicyclic graphs on n vertices having girth 4, the algebraic connectivity 
is uniquely maximized by D4y—4. @ 


[FKP03b] Fix g > 5. There is an N such that ifn > N, then 
over the class of unicyclic graphs on n vertices having girth g, 
the algebraic connectivity is uniquely maximized by Dg,n—¢. 
[Kir03] For each real number r > 0, there is a sequence of 
graphs Gx with distinct algebraic connectivities, such that 
a(G;,) converges monotonically to r as k — oo. 


FIGURE 36.3 The graph D43. 


Examples: 


1. 


2. 
3. 


The graph C,,3 is shown in Figure 36.2; its algebraic connectivity 
is approximately 0.3249. 


The graph D4, is shown in Figure 36.3. 


[FK98] The algebraic connectivity of D3,,-3 is 1, and it is an 
eigenvalue of multiplicity n — 3 of the corresponding Laplacian 
matrix. 


Consider the graph G pictured in Figure 36.4. Its algebraic 
connectivity is 2, so that from Fact 5 above, we deduce that FIGURE36.4 The graph G for 
1 < ®(G) < 2/2. It turns out that the value of ®(G) is 3. Example 4 


36.3 Algebraic Connectivity for Trees 


The term graph means simple graph in this section. 


Definitions: 


Let T be a tree, and let y be a Fiedler vector for T. A characteristic vertex of T is a vertex i satisfying 
one of the following conditions: 


I y; = 0, and vertex i is adjacent to a vertex j such that y; 4 0. 


II There is a vertex j adjacent to vertex i such that yjy; < 0. 


A tree is called type I if it has a single characteristic vertex, and type II if it has two characteristic vertices. 
Let T be a tree and suppose that v is a vertex of T. The bottleneck matrix of a branch of T at v is the 
inverse of the principal submatrix of the Laplacian matrix corresponding to the vertices of that branch. 
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A branch at vertex v is called a Perron branch at v if the Perron value of the corresponding bottleneck 
matrix is maximal among all branches at v. 

Suppose that T is a type I tree with characteristic vertex i. A branch at i is called an active branch if, 
for some Fiedler vector y, the entries in y corresponding to the vertices in the branch are nonzero. 

Suppose that n > d+ 1. Denote by P (n, d) the tree constructed as follows: Begin with the path P44, 0n 
vertices 1,... ,d + 1, labeled so that vertices 1 and d + 1 are pendent, while for eachi = 2,... ,d, vertexi 
is adjacent to vertices i — 1 andi + 1, then append n — d — 1 isolated vertices at vertex (4 + 1 of that path. 

Let T(k,1,d) be the treeon n = d+k-+1 vertices constructed from the path Py by appending k isolated 
vertices at one end vertex of Py, and appending / isolated vertices at the other end vertex of Pj. 


Facts: 


1. The bottleneck matrix for a branch is the inverse of an irreducible M-matrix, and so is entrywise 
positive (see Chapter 9.5 and/or [KNS96]). 

[Fie89] If T is a tree on n vertices, then 2(1 — cos(¥)) < a(T); equality holds ifand only ifG = P,,. 
[Mer87] If T is a tree on n > 3 vertices, then a(G) < 1; equality holds if and only if G = K,,-1. 
[GMS90] If T is a tree with diameter d, then a(T) < 2(1 — cos(z77)). 

[Fie75b] Let T be a tree on vertices labeled 1,... ,n, and suppose that y is a Fiedler vector of T. 


a ee PS 


Then exactly one of two cases can occur. 


* There are no zero entries in y. Then T contains a unique edge {i, j} such that yj; < 0 < y;. As we 
move along any path in T that starts at i and does not contain j, the corresponding entries in y 
are positive and increasing. As we move along any path in T that starts at 7 and does not contain 
i, the corresponding entries in y are negative and decreasing. In this case T is type II. 


* The vector y has at least one zero entry. Then there is a unique vertex i of T such that y; = 0 and 
i is adjacent to a vertex j such that y; 4 0. As we move along any path in T that starts at vertex 
i, the corresponding entries in y are either increasing, decreasing, or identically zero. In this case 
T is type I. 


6. [KNS96] Let T be a tree and y be a Fiedler vector for T. Let P be a path in T that starts at a 
characteristic vertex i of T, and does not contain any other characteristic vertices of T. If, as we 
move along P away from i the entries of y are increasing, then they are also concave down. If, as 
we move along P away from i the entries of y are decreasing, then they are also concave up. 

7. [Mer87] Let T be a tree. Then each Fiedler vector for T identifies the same vertex (or vertices) 
as the characteristic vertex (or vertices). Consequently, the type of the tree is independent of the 
particular choice of the Fiedler vector. 

8. [Fie75a] If T is a type II tree, then a(T) is a simple eigenvalue. 

9. [KNS96] A tree T is type I if and only if at some vertex 7, there are two or more Perron branches. 
In that case, i is the characteristic vertex of T, w(T) is the reciprocal of the Perron value of the 
bottleneck matrix of a Perron branch at 7, and ([KF98]) the multiplicity of w(T) is one less than 
the number of Perron branches at i . 

10. [KNS96] A tree T is type IL ifand only if there is a unique Perron branch at every vertex. In this case, 
the characteristic vertices of T are the unique adjacent vertices i, j such that the Perron branch at 
vertex i is the branch containing vertex j, and the Perron branch at vertex j is the branch contain- 
ing vertex i. Letting B; and B; denote the bottleneck matrices for the Perron branches at vertices 
i and j, respectively, d!y € (0,1) such that p(B; — yJ) = p(B; — (1 — y)J), and the common 
Perron value for these matrices is 1/a(T). 

11. The following is a consequence of results in [GM87] and [KNS96]. Suppose that T isa type I tree with 
characteristic vertex i; then a branch at i is an active branch if and only if it is a Perron branch at 7. 


12. [FK98] Among all trees on n vertices with diameter d, the algebraic connectivity is maximized by 
P(n,d). 
13. [FK98] Let T bea tree on n vertices with diameter d. Then a(T) > a(T([ nated L[ wt ],d—1)), 


with equality holding if and only if T = T(["-*], [>"],d - 1). 
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Examples: 


1. [GM90] The algebraic connectivity of T(k, 1,2) is the smallest root of the polynomial x? — (k + 
1+4)x? + (2k +21 + kl +5)x —(k +142). 

2. Let T be the tree constructed as follows: At a single vertex v, append k > 2 copies of the path P, 
and/ > 0 pendent vertices. For each branch at v consisting of a path on two vertices, the bottleneck 


1 : ‘ Sern 
| , which has Perron value 34-V5 | while each branch at v consisting 


: : Z 
matrix can be written as ; l 5) 


of a single vertex has bottleneck matrix equal to [1]. Then a(T) = ae and is an eigenvalue 


of multiplicity k — 1 of the corresponding Laplacian matrix. In particular, T is a type I tree with 
characteristic vertex v. 

3. [FK98] If d > 4 is even, a(P(n,d)) = 2(1 — cos( 47) and is a simple eigenvalue of the corre- 
sponding Laplacian matrix. 

4. Consider the tree T(2,2,2) shown in Figure 36.5. At both of 
its nonpendent vertices, the bottleneck matrix for the corre- 


2d 
sponding Perron component can be written as |1 2 1). 
1 11 
2141 FIGURE 36.5 The tree T(2, 2,2). 
It follows that 1/a(T) = pe 1 2 1) -1/2J | = (+ 


111 


17)/4. Hence, the algebraic connectivity for T (2, 2, 2) is (5 — 


17)/2, and it is a type I] tree, with the two nonpendent 


34V17 34717 1 3+V17 3+V17 1] 
4? 4.07? 4? 4? 


vertices as the characteristic vertices. [ T is a Fiedler vector. 


5. [GM87] Consider the tree T shown in Figure 36.6; this is a type I tree with characteristic vertex v. 
The numbers above the vertices are the vertex numbers and the numbers below are (to four decimal 
places) the entries in a Fiedler vector for T. The bottleneck matrix for the branch at v on 5 vertices 

; eg 


13 1 2 #1 
can be written as |2 1 2 1 1/4 , while that for the branch at v on 4 vertices can be written 

lt 324, A 2s all 
Ms oh de dud 

Se 2 2-01 

F935. 9 1 wh 3 thors <n 

a ar oe | The algebraic connectivity is the smallest root of the polynomial x’? — 6x“ +8x— 1, 

Lk A 


and is approximately 0.1392. 


1.6617 1.4304 —2.1617 


FIGURE 36.6 The tree T in Example 5. 


Algebraic Connectivity 36-7 


36.4 Fiedler Vectors and Algebraic Connectivity 
for Weighted Graphs 


The term graph means simple graph in this section. 


Definitions: 


Let G = (V, E) bea graph on n > 2 vertices, and suppose that for each edge e € E we have an associated 
positive number w(e). 


* Then w(e) is the weight of e. 
* The function w : E > Rt? isa weight function on G. 
* The graph G, together with the function w : E > Rt, is a weighted graph, and is denoted Gy. 


The Laplacian matrix L(G,,) for the weighted graph G, is the n x n matrix L(Gy) = [I;,;] such 
that /;; = 0 if vertices i and j are distinct and not adjacent in G, ];,; = —w(e) ife = {i,j} € E, and 
l,; = >2 w(e), where the sum is taken over all edges e incident with vertex i. 

Throughout this chapter, we only consider Laplacian matrices for weighted graphs on at least two 
vertices. The notation L(G,,) for the Laplacian matrix of the weighted graph G,, should not be confused 
with the notation for the line graph introduced in Section 28.2. 

Let G,, be a weighted graph on n vertices, and denote the eigenvalues of L(G,,) by 0 = wu, < wo < 

. < fy. The algebraic connectivity of G,, is 42 and is denoted a(G,,). 

Let Gy be a weighted graph. A Fiedler vector for G,, is an eigenvector of L(G,,) corresponding to the 
eigenvalue a(G,,). 

Let G,, be a weighted graph and y be a Fiedler vector for G,,. For each vertex i of Gy, the valuation of 
i is equal to the corresponding entry in y. A vertex is valuated positively, negatively, or zero accordingly, 
as the corresponding entry in y is positive, negative, or zero. 


Facts: 
1. [Fie75b] Let G,, be a weighted graph and let y be a Fiedler vector for G,,. Then exactly one of the 
following holds. 


* Case A—There is a single block By in Gy containing both positively valuated vertices and 
negatively valuated vertices. For every other block of G,,, the vertices are either all valuated 
positively or all valuated negatively or all valuated zero. Along any path in G that starts at a vertex 
k € Bo and contains at most two cut-vertices from each block, the valuations of the cut-vertices 
form either an increasing sequence, a decreasing sequence, or an all zero sequence, according as 
Ve > 0, ¥~ < 0, or yy = 0, respectively. In the case where yz = 0, then each vertex on that path 
is valuated zero. 


* Case B — No block in G,, has both positively and negatively valuated vertices. Then there is a 
unique vertex i of G,, having zero valuation that is adjacent to a vertex j with nonzero valuation. 
The vertex i is a cut-vertex. Each block of G, contains (with the exception of i) only vertices 
of positive valuation, or only vertices of negative valuation, or only vertices of zero valuation. 
Along any path that starts at vertex 7 and contains at most two cut-vertices from each block, the 
valuations of the cut-vertices form either an increasing sequence, a decreasing sequence, or an 
all zero sequence. Any path in G,, that contains both positively and negatively valuated vertices 
must pass through vertex i. 


2. [KF98] Let G,, be a weighted graph. If case A holds for some Fiedler vector, then case A holds for 
every Fiedler vector and, moreover, every Fiedler vector identifies the same block of Gy, having 
vertices of both positive and negative valuations. Similarly, if case B holds for some Fiedler vector, 
then case B holds for every Fiedler vector and, moreover, every vector identifies the same vertex i 
that is valuated zero and is adjacent to a vertex with nonzero valuation. 
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[KF98] Suppose that for a weighted graph G,,, case A holds, and let Bo be the unique block of Gy, 
containing both positively and negatively valuated vertices. Fix an edge e € Bo, and let w be the 
weighting of G formed from w by replacing w(e) by t, where 0 < t < w(e). Then for the weighted 
graph Gy, case A also holds, and Bo is still the block of G containing both positively and negatively 
valuated vertices. Similarly, if By — e is a block of G — e, then case A still holds for the weighting 
w of G — e arising from setting w(f) = w(f) for each f € E\{e}, and Bo — e is still the block 
containing both positively and negatively valuated vertices. 

[KF98] Suppose that G is a graph, that w is weight function on G, and consider the weighted 
graph G,,. Suppose that B is a block of G,, that is valuated all nonnegatively or all nonposi- 
tively by some Fiedler vector. Let G,, be the weighted graph formed from G,, by either adding 
a weighted edge to B, or raising the weight of an existing edge in B, and denote the modified 
block by B. Then every Fiedler vector for G,, valuates the vertices of B all nonnegatively, or all 
nonpositively. 


. [FN02] Suppose that G,, is a weighted graph on n vertices with Laplacian matrix L(G, ). For each 


nonempty subset U of the vertex set V, let L(G,,)[U] denote the principal submatrix of L(G,,) 
on the rows and columns corresponding to the vertices in U, and let t(L(G,,)[U]) denote its 
smallest eigenvalue. Then a(G,,) > min{max{t(L(G,,)[U]), t(L(Gy)[V\ U])}|U C V,0 < |U| 


[KN04] Suppose that G = (V, E) is a graph, that w is weight function on G, and consider the 
weighted graph G,,. Fix an edge e = {i, j} € E. For each t > w(e), let w; be the weight function 
on G constructed from w by setting the weight of the edge e to f. 


* If some Fiedler vector y for G,, has the property that y; = y;, then for any t > w(e),a(Gy,) = 
a(G,,). In particular, ifa(G,,) is an eigenvalue of multiplicity at least two, then a(Gy,) = a(G,) 
for all t > w(e). 

* If a(G,,) is a simple eigenvalue, then Af) > 0 such that for each t € [0,%),a(Gy,) is a twice 
differentiable function of t, with 0 < —— < 2and eal) < 0. Further, let y(t) denote 

a Fiedler vector for Gy, of norm 1 that is analytic for t € [0,t)); then |(y(t)); — (y(t))j| is 

nonincreasing for t € [0, fo). 


Examples: 


1. 


2: 


Consider the weighting of K,, in which each edge has weight 1. Then any vector that is orthogonal 
to 1,, is a Fiedler vector. 

Consider the weighting of K,,-1 in which edge has weight 1. If i and j are pendent vertices of 
Ky n-1, then e; — e; is a Fiedler vector. 

Consider the weighting of the path P,, in which each edge has weight 1, and label the vertices 
of P, so that vertices 1 and n are pendent, while for each i = 2,...,n — 1, vertex i is adja- 
cent to vertices i — 1 andi + 1. Then the vector y with y; = cos((2i — 1)m/(2n)) is a Fiedler 
vector. 

Suppose that a,b > 0, and let T,, be the weighting of the path P, arising by assigning the pendent 
edges weight a, and the nonpendent edge weight b. The corresponding Laplacian matrix can be 


a 0 —a 0 
: 0 a 0 —a : . 
written as , which has eigenvalues 0,a + b + Ja? + b?, and 2a. In 
—a 0 atb —b 
0 -a —b atb 
a 
—a 


articular, a(T,) = a+b — Ja? + b?, with corresponding Fiedler vector : 
ae V®+e —b 


b— Ja? +b? 
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5. Figure 36.7 shows a weighted graph G,,, where the parameters a, b,c 
are the weights of the edges, and where the vertices are labeled 


1,... 4, as indicated in that figure. The corresponding Laplacian 
matrix is L(G,,) = 

b 0 0 —b 

0 atc c 


a 
. The ei | fL(G 0, 
0 —c atc —a SelpenvaluesOn RUG y) ate FIGURE 36.7 Theweighted graph 


G,, in Example 5. 


—b —a —a 2at+b 
+ /9a? dab F452 : _ 9a? dab 4p? 
a+2c,and Batibte/ 9a" sab HAP” so that «(G,,) = min{a+ 2c, Sedan va Py Ifa(Gy) =a4+2c, 
3a+J/9a?2—4ab+4b2 
2 
/9a2—4ab bab? 
Ly : 1A: 3at+2b—/9a?—4ab+4b? adept “a Sebt4 
then isa Fiedler vector for G,,, while ifa(G,,) = 5 , then 
—1 a+2b+/9a?—4ab+4b? 
4 


b-a 


is a Fiedler vector for Gy. 


36.5 Absolute Algebraic Connectivity for Simple Graphs 


The term graph means simple graph in this section. 


Definitions: 


Let G = (V, E) bea graph onn > 2 vertices. Let W denote the set of all nonnegative weightings of G with 
the property that for each weighting w, we have >... We = |E| (observe that here we relax the restriction 
that each edge weight be positive, and allow zero weights). The absolute algebraic connectivity of G is 
denoted by &(G), and is given by &(G) = max{a(G,)|w € W}. 

Let T = (V, E) bea tree. For each vertex u of T, define S(u) = rev (d(u, k)). 


Facts: 


1. [Fie90] For a graph G, &(G) = 0 if and only if G is disconnected. 

2. [Fie90] If G is a graph and Q is its automorphism group, then d(G) = max{a(Gy)|w € Wo}, 
where Wo is the subclass of W consisting of weightings for which equivalent edges in {2 have the 
same weight. 

3. [KP02] If G is a graph with m edges, and H is the graph formed from G by adding an edge not 
already present in G, then ma (G) < &(H). 


4. [Fie90] Let T be a tree on n vertices. Then one of the following two cases occurs: 


n—1 


* There is a vertex v of T such that for any vertex k #4 v, S(v) < S(k)—n. In this case, d(T) = 3G)" 


4n(n—1) 
4nS(u)—(n+S(u)—S(v))?* 


* There is an edge {u, v} of T such that |S(u) — S(v)| < n. In this case, &@(T) = 


Weightings of trees that yield the maximum value @ are also discussed in [Fie90]. 
5. [Fie93] If T is a tree on n vertices, then wnt) < &(T) < 1. Equality holds in the lower bound if 
T = P,, and equality holds in the upper bound if T = Ky y,—1. 


6. [KP02] Suppose that G is a graph on n > 7 vertices that has a cut-vertex. Then &(G) < 


m—3n+4 4 e : 
( 3 ) 1 SP EEN aN 5): Further, the upper bound in attained by the graph 


formed by appending a pendent vertex at a vertex of K,,_; a weighting of that graph that yields the 
maximum value @ is also given in [KP02]. 


36-10 Handbook of Linear Algebra 


FIGURE 36.8 The tree P(9,5). 


FIGURE 36.9 The tree T (2, 3, 4). 


Examples: 
1. The absolute algebraic connectivity of K, is n. 
2. Ifm < nand2 <n, then d(Km,) = m. 
2 
3. [KP02] Let e be an edge of K,. Then &(K,, — e) = es 
4. Consider the tree P (9,5) shown in Figure 36.8; we have S(v) = 22, and S(v) < S(k) — 9 for any 


vertex k 4 v, so that the first case of Fact 4 above holds. Hence, &(P(9,5)) = 4. 
5. Consider the tree T (2, 3, 4) pictured in Figure 36.9. We have S(u) = 41 and S(v) = 36, so that the 
second case of Fact 4 above holds. Hence, &(T (2, 3,4)) = z. 


36.6 Generalized Laplacians and Multiplicity 


The term graph means simple graph in this section. 


Definitions: 


Let G = (V, E) bea graph on n > 2 vertices. A generalized Laplacian of G is a symmetric n x n matrix 
M with the property that for each i # j, mj; < Oif {i,j} €¢ E and m;,; = Oif {i, j} ¢ E. 

We only consider generalized Laplacian matrices for graphs on at least two vertices. 

For any symmetric matrix B, let t(B) denote its smallest eigenvalue, let 4(.B) denote its second smallest 
eigenvalue, and let A(B) denote its largest eigenvalue. 

Let G be a graph, and let v be a vertex of G. Suppose that M is a generalized Laplacian matrix for G. 
Let C;,... , Cz denote the components of G — v; a component C; is called a Perron component at v if 
t(M[Cj]) is minimal among all t(M[C;]),i = 1,... ,k. 


Facts: 


1. [BKPO1] Let G bea connected graph and let v be a cut-vertex of G. Suppose that M is a generalized 
Laplacian matrix for G. If there are p > 2 Perron components at v, say C),... ; Cp, then w(M) = 
t(M[C,]). Further, 4(M) is an eigenvalue of M of multiplicity p — 1. 

2. [BKP01] Let G be a connected graph, and let v be a cut-vertex of G. Let M be a generalized 
Laplacian matrix for G. Suppose that there is a unique Perron component Ao at v; denote the 
other connected components at v by B,,... , By, and set Ay = G\ Apo. Let z be an eigenvector of 
M corresponding to t(M), and let zo, z be the subvectors of z corresponding to the vertices in 
Ao, Aj, respectively. For eachi = 1,... ,k, denote the principal submatrix of M corresponding to 
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the vertices of B; by M[B;]. Let S = (M[B,])"! ©... ® (M[Bx])7! @ [0], and let My denote the 
principal submatrix of M corresponding to the vertices of Ao. There is a unique y > 0 such that 
(My! —yZ2%!) = A(S+yzz,") = 1/(M). Further, the multiplicity of u(M) as an eigenvalue 


of M coincides with the multiplicity of 1/j4(M) as an eigenvalue of My '_ yaoz9!. 


Examples: 


—1 0 0 2 


—2 0 0 0 
Perron components at vertex 1, induced by the vertex sets {2, 3}, {4}, and {5}. We have w(M) = 2, 
which is an eigenvalue of multiplicity two. 


0 

1. Consider the generalized Laplacian matrix M = |—1 -1 3 0 _ 0O|. There are three 
0 
2 


3 1 1 1 2 
—1 2-11 0 0 
2. Consider the generalized Laplacian matrix M= |—1 —1 2 0 0] .The unique Perron 
=a. 0 0 2 0 
=) 0 0 0 2 
component at vertex 1 is induced by the vertex set {2,3}. We have w(M) = catalDD which is a 


simple eigenvalue. 
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Calculations are subject to errors. There may be modeling errors, measurement errors, manufacturing 
errors, noise, equipment is subject to wear and damage, etc. In preparation for computation, data must 
often be perturbed by rounding it to fit a particular finite precision, floating-point format. Further errors 
may be introduced during a computation by using finite precision arithmetic and by truncating an infinite 
process down to a finite number of steps. 

This chapter outlines aspects of how such errors affect the results of a mathematical computation with 
an emphasis on matrix computations. Topics include: 


* Vector and matrix norms and seminorms that are often used to analyze errors and express error 
bounds. 

* Floating point arithmetic. 

* Condition numbers that measure how much data errors may affect computational results. 

* Numerical stability, an assessment of whether excessive amounts of data may be lost to rounding 
errors during a finite precision computation. 


For elaborative discussions of vector and matrix norms, consult [HJ85]. See [Ove01] for a textbook 
introduction to IEEE standard floating point arithmetic. Complete details of the standard are available 
in [IEEE754] and [IEEE854]. Basic concepts of numerical stability and conditioning appear in numerical 
linear algebra books, e.g., [GV96], [Ste98], [TB97], and [Dat95]. The two particularly authoritative books 
on these topics are the classical book by Wilkinson [Wil65] and the more recent one by Higham [Hig96]. 
For perturbation analysis, see the classic monograph [Kat66] and the more modern treatments in [SS90] 
and [Bha96]. 


37-1 


37-2 Handbook of Linear Algebra 


The set C” = C"™! is the complex vector space of n-row, 1-column matrices, and R” = R™! is the real 
vector space of n-row, 1-column matrices. Unless otherwise specified, F” is either R” or C”, x and y are 
members of F”, anda € Fisascalara € Rora € C, respectively. For x € R", x! is the one row n-column 
transpose of x. For x € C”, x* is the one row n-column complex-conjugate-transpose of x. A and B are 
members of F”*". For A € R”*", A* € R”*” is the transpose of A. For A € C”*", A* € C”*” is the 
complex-conjugate-transpose of A. 


37.1 Vector Norms 


Most uses of vector norms involve R” or C", so the focus of this section is on those vector spaces. However, 
the definitions given here can be extended in the obvious way to any finite dimensional real or complex 
vector space. 

Let x,y € F” anda € F, where F is either R or C. 


Definitions: 


A vector norm is a real-valued function on F” denoted ||x|| with the following properties for all x, y € F" 
and all scalarsa@ € F. 


* Positive definiteness: ||x|| > 0 and ||x|| = 0 if and only if x is the zero vector. 
* Homogeneity: ||ox|| = |a|||x||. 
* Triangle inequality: ||x + y|| < ||x|| + llyll. 


For x = [X), X2,X3,... 5 X,]* € F", the following are commonly encountered vector norms. 


¢ Sum-norm or 1-norm: ||x||; = |x,| + |x2| + --- + |x,|. 
* Euclidean norm or 2-norm: ||x||. = /|x1|? + |a2|2 +--- + [xnl?. 
¢ Sup-norm or oo-norm: ||x||,o = max |x;|. 

l<i<n 


* Hélder norm or p-norm: For p > 1, |x| p = (lxi]? + +--+ lx,|P)?. 


If || - || is a vector norm on F” and M ¢€ F"”*" is a nonsingular matrix, then ||y||a = ||My|| is an 
M-norm or energy norm. (Note that this notation is ambiguous, since || - || is not specified; it either 
doesn’t matter or must be stated explicitly when used.) 

A vector norm || - || is absolute if for all x € F”, || |x| || = ||x|], where |[x1,...,x.]*| = [lxil,..-5 [xn ]*. 

A vector norm || - || is monotone if for all x, y € F”, |x| < |y| implies ||x|| < |ly|]. 

A vector norm || - || is permutation invariant if || Px|| = ||x|| for allx € R” and all permutation matrices 
PeR™", 

Let || - || be a vector norm. The dual norm is defined by |ly||? = maxxzo a 

The unit disk corresponding to a vector norm || - || is the set {x € F” | ||x|| < 1}. 

The unit sphere corresponding to a vector norm || - || is the set {x € F” | |x|] = 1}. 

Facts: 


For proofs and additional background, see, for example, [HJ85, Chap. 5]. 
Let x,y € F” anda € F, where F is either R or C. 


1. The commonly encountered norms, || - |[1, | - ll) ll - lloo» Il - lp, are permutation invariant, absolute, 
monotone vector norms. 

2. If M € F"*" isa nonsingular matrix and || - || isa vector norm, then the M-norm || - || is a vector 
norm. 

3. If || - || isa vector norm, then | {|x| — |lyll | < llx— yl. 


4. A sum of vector norms is a vector norm. 


5. lim x, = x, ifand only ifin any norm lim ||x; — x,|| = 0. 
k->0oo k>oo 
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6. Cauchy—Schwartz inequality: 

(a) |x*yl < [Ixllollylle. 

(b) |x*y| = ||xll2|lyll2 ifand only if there exist scalars w and f, not both zero, for which ax = By. 
7. Hélder inequality: If p > 1 andq > 1 satisfy Fs + ; = 1, then |x*y| < ||xIl>p llyll,- 


8. If || - || isa vector norm on F”, then its dual || - ||? is also a vector norm on F", and || - ||?” = || - |]. 

9. If p > Oandq > O satisfy ++ 1 = 1, then || []2 = [| |[q- In particular, || - | = || Il. Also, 
MP =U Mhoo + 

10. If || - || isa vector norm on F", then for any x € F”, |x*y| < ||x|l lly|]?. 


11. A vector norm is absolute if and only if it is monotone. 

12. Equivalence of norms: All vector norms on F” are equivalent in the sense that for any two vector 
norms || - ||,, and || - |], there constants w > 0 and B > 0 such that for all x € F", a |[x\|,, < ||xll, < 
A||x\|,,. The constants @ and 6 are independent of x but typically depend on the dimension n. 

In particular, 


(a) xll2 < xl < Vallxll2. 
(b) [IXlloo S [Ixll2 S Vil|xlloo. 


(c) [Xlloo S [xl < n1Xlloo. 


13. Aset D C F” is the unit disk of a vector norm if and only if it has the following properties. 
(a) Point-wise bounded: For every vector x € F” there is a number 6 > 0 for which 6x ¢ D. 
(b) Absorbing: For every vectorx € F” there isa number t > 0 for which |a| < t impliesawx € D. 


(c) Convex: For every pair of vectors x, y € D and every number t,0 < t < 1,tx+(1—1t)ye D. 


Examples: 
1. Let x = [1,1,—2]*. Then ||x|], = 4, ||xl]x = /12 + 12 + (—2)? = V6, and ||x|loo = 2. 


Balle 


. Using the 1-norm, = 6. 


1 
2. Let M= 
: : 4 


M | 1 


37.2 Vector Seminorms 


Definitions: 
A vector seminorm is a real-valued function on F", denoted v(x), with the following properties for all 
x, y € F” and all scalars a ¢€ F. 

1. Positiveness: v(x) > 0. 

2. Homogeneity: ||ox|| = |a|||x||. 

3. Triangle inequality: ||x + y|| < [|x|] + llyll. 


Vector norms are a fortiori also vector seminorms. 


The unit disk corresponding to a vector seminorm || - || is the set {x € F” | v(x) < 1}. 
The unit sphere corresponding to a vector seminorm || - || is the set {x € F” | v(x) = 1}. 
Facts: 


For proofs and additional background, see, for example, [HJ85, Chap. 5]. 
Let x,y € F” anda ¢€ F, where F is either R or C. 


1. 
2. 


(0) =0 


v , 
v(x —y) = |vx) — v(y) |. 
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3. Asum of vector seminorms is a vector seminorm. If one of the summands is a vector norm, then 
the sum is a vector norm. 
4. Aset D C F” is the unit disk of a seminorm if and only if it has the following properties. 


(a) Absorbing: For every vector x € F” there isanumber t > 0 for which |a| < t impliesax € D. 


(b) Convex: For every pair of vectors x, y € D and every number t,0 < t < 1,tx+(1—1t)ye D. 


Examples: 
1. For x = [x1,%2,x3,...5Xn]! € F", the function v(x) = |x| is a vector seminorm that is not a 
vector norm. For n > 2, this seminorm is not equivalent to any vector norm || - ||, since ||e2|| > 0 


but v(e.) = 0, for er = [0,1,0,...,0]7. 


37.3. Matrix Norms 


Definitions: 


A matrix norm is a family of real-valued functions on F”™” for all positive integers m and n, denoted 
uniformly by || A|| with the following properties for all matrices A and B and all scalars a € F. 


* Positive definiteness: || A|| > 0; || Al] = 0 only if A = 0. 

* Homogeneity: || Al] = |@||| All. 

* Triangle inequality: || A+ B|| < || All + || B||, where A and B are compatible for matrix addition. 
* Consistency: || AB|| < || Al] || B||, where A and B are compatible for matrix multiplication. 


If || - || is a family of vector norms on F" for n = 1, 2,3,... , then the matrix norm on F”*" induced by 
: Ax ; 
(or subordinate to) || - || is || Al] = max,~o a Induced matrix norms are also called operator norms 
x 


or natural norms. The matrix norm || A||, denotes the norm induced by the Holder vector norm ||x|| p. 
The following are commonly encountered matrix norms. 


m 


* Maximum absolute column sum norm: || A||; = max S |a;;|. 
I<jsn¢ 
i=1 


* Spectral norm: || All, = /(A* A), where p(A* A) is the largest eigenvalue of A* A. 


* Maximum absolute row sum norm: || Al|,. = max S- |a;;\. 
l<i<m 
Sis 


* Euclidean norm or Frobenius norm: || A|| - = 


Let M = {M, € F"*" :n > 1} bea family of nonsingular matrices and let || - || be a family of vector 
norms. Define a family of vector norms by ||x||4 for x € F" by |x| = ||©!M,x||. This family of vector 
norms is also called the M-norm and denoted by || - ||, . (Note that this notation is ambiguous, since 
|| - || is not specified; it either does not matter or must be stated explicitly when used.) 

A matrix norm || - || is minimal if for any matrix norm || - ||,, || All, < || All for all A e¢ F”*” implies 
Ill, = +I 


A matrix norm is absolute if as a vector norm, each member of the family is absolute. 
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Facts: 


For proofs and additional background, see, for example, [HJ85, Chap. 5]. Let x,y € F”, A, B € F”*", and 
a € F, where F is either R or C. 


Ie 


15. 


16. 


17. 
18. 


A matrix norm is a family of vector norms, but not every family of vector norms is a matrix norm 
(see Example 2). 


. The commonly encountered norms, || - ||1, || - [l25 |] - [loos || - || z=, and norms induced by vector norms 


are matrix norms. Furthermore, 


(a) || All; is the matrix norm induced by the vector norm || - ||;. 
(b) ||All2 is the matrix norm induced by the vector norm || - ||2. 
(c) ||Alloo is the matrix norm induced by the vector norm || - ||,o- 


(d) ||All¢ is not induced by any vector norm. 


(e) If M = {M,} is a family of nonsingular matrices and || - || is an induced matrix norm, then 
for A € F"*", || All = ||MnAM,, "|. 


If || - || is the matrix norm induced by a family of vector norms || - ||, then || I,|| = 1 for all positive 
integers n (where I, is the n x n identity matrix). 
If || - || is the matrix norm induced by a family of vector norms || - ||, then for all A € F”*” and all 


x € F", || Ax|| < |All IIx). 
For all A € F”*" and all x € F”, || Ax||- < || Alle ||xllo. 


ll - lao Il loos Il Ile are absolute norms. However, for some matrices A, |||Al |l2 4 ||All2 
(see Example 3). 
A matrix norm is minimal if and only if it is an induced norm. 

. All matrix norms are equivalent in the sense that for any two matrix norms || - ||,, and || - ||), there 


exist constants a > 0 and 6 > 0 such that for all A € F”*", a|[All, < ||Ally) < BIAll,. The 
constants a and £ are independent of A but typically depend on n and m. In particular, 


(a) FallAlloo S lAlla S$ ml Alloo- 
(b) Allz < l|Allr < Jal Alla. 
(c) sallAlh < lAlle < VallAlh. 


lAll2 = VITATIIAll oo. 

. |ABlle < |All || Bll2 and || AB||z < ||Allz||Bllz whenever A and B are compatible for matrix 
multiplication. 

. ||All2 < || Alle and || All2 = || Alle if and only if A has rank less than or equal to 1. 


. If A= xy* for some x € F” andy € F”, then || Al|2 = || Allz = |Ixllallyll2. 
. |All = ||A*ll2 and || Ally = ||A* lle. 
. IfU € F"*" is a unitary matrix, i.e., if U* = U~!, then the following hold. 


(a) |Ullz = land |U lp = Jn. 
(b) If A ¢ F”*", then || AU||2 = || Allz and || AU]lz = || Alle. 


(c) IfAe F”™, then |[U All, = |] Allz and ||U Alle = || Alle. 


For any matrix norm || - || and any A € F”*", p(A) < |All, where o(A) is the spectral radius of 
A. This need not be true for a vector norm on matrices (see Example 2). 

Forany A € F"*" andeé > 0, there exists a matrix norm || - || such that || Al] < p(A)+e. A method 
for finding such a norm is given in Example 5. 

For any matrix norm || - || and A € F"*", limpsool| Af ||/* = p(A). 

For A € F"*", limy_,..A* = 0 if and only if p(A) < 1. 
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Examples: 


[£2 
lL A= | ws the \| All: = 6, || Alloo = 7 l|All2 = 15 + V221, and || Allp = V30. 


3 
2. The family of matrix functions defined for A € F”*" by 
v(A) = max aj; 
l<i<m 


l<j<n 


; ; ‘ 2 : 1 1 
is not a matrix norm because consistency fails. For example, if J = | j ,thenv(J?) =2>1= 


1 
v(J )v(J ). Note that v is a family of vector norms on matrices (it is the 00 norm on the n?-tuple of 
entries), and v(J) =1<2= (J). 


3 4 


3. IfA= ae 5s 


Jot I] Allz = 5 but |] |A] lz = 7. 


4. If Ais perturbed by anerror matrix E and U isunitary (i.e., U* = U~'), then U(A+E) = UA+UE 
and ||UE||2 = ||£||2. Numerical analysts often use unitary matrices in numerical algorithms 
because multiplication by unitary matrices does not magnify errors. 

5. Given A € F"*" andeé > 0, weshowhowan M-norm can be constructed such that || All|, < p+e, 
where p is the spectral radius of A. The procedure below determines M,, where A € F”*". The 


—38 13 52 
procedure is illustrated with the matrix A= | 3 Q  —4] and with e = 0.1. The norm used 
—30 10 41 


to construct the /-norm will be the 1-norm; note the 1-norm of A = 97. 


(a) Determine p: The characteristic polynomial of Ais p4(x) = det(A—xI) = x°—3x?+3x-1= 
(x —1%,sop=1. 

(b) Find a unitary matrix U such that T = U AU* is triangular. Using the method in Example 5 
of Chapter 7.1, we find 


1 6 3 
a oa 0.316228 0.744208  —0.588348 
U=|va ~ve VE | & |0.948683 —0.248069 0.196116 | and 
- 5 0. 0.620174 0.784465 
0 4/5 my /= 
13 13 
1 0 265 1 0 16.1245 
T=U*AU=|0 1 26/10| © [0 1 82.2192]. 
00 1 ed 


(c) Find a diagonal matrix diag(1,@,@7,...,a"~') such that || DT D7!||, < p + (this is always 
possible, since limy—o || DTD ||; = p). 
1 0 0.0000161245 


In the example, fora = 1000, DTD"! © |0 1  0.0822192 | and ||DTD™'|, ~ 
0 1 


Oo 


1.08224 < 1.1. 
(d) Then ||DU*AUD™!||,; < p + €. That is, || Ally < 2.1, where M3; = DU* 
0.316228 0.948683 0. 
XS | 744.208 —248.069 620.174 
—588348. 196116. 784465. 
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37.4 Conditioning and Condition Numbers 


Data have limited precision. Measurements are inexact, equipment wears, manufactured components meet 

specifications only to some error tolerance, floating point arithmetic introduces errors. Consequently, the 

results of nontrivial calculations using data of limited precision also have limited precision. This section 

summarizes the topic of conditioning: How much errors in data can affect the results of a calculation. 
(See [Ric66] for an authoritative treatment of conditioning.) 


Definitions: 


Consider a computational problem to be the task of evaluating a function P : R" + R” ata nominal data 
point z € R", which, because data errors are ubiquitious, is known only to within a small relative-to-||z|| 
error €. 

If z € F" is an approximation to z € F"”, the absolute error in Z is ||z — 2|| and the relative error in Z is 
||z — || /|lz||. If z = 0, then the relative error is undefined. 

The data z are well-conditioned if small relative perturbations of z cause small relative perturbations of 
P(z). The data are ill-conditioned or badly conditioned if some small relative perturbation of z causes a 
large relative perturbation of P(z). Precise meanings of “small” and “large” are dependent on the precision 
required in the context of the computational task. 

Note that it is the data z—not the solution P (z) — that is ill-conditioned or well-conditioned. 

If z 4 0 and P(z) ¥ 0, then the relative condition number, or simply condition number cond(z) = 
cond p(z) of the data z € F” with respect to the computational task of evaluating P(z) may be defined 
as 


| P(z + 6z) — =e) ( I|z| ) 
I| P(z)lI I|5z|| 


condp(z) = lim sup ( ||dz|| < | : (37.1) 


Sometimes it is useful to extend the definition to z = 0 or to an isolated root of P (z) by condp(z) = lim sup 
x—>Z 
cond p(x). 


Note that although the condition number depends on P and on the choice ofnorm, cond(z) = cond p(z) 
is the condition number of the data z—not the condition number of the solution P(z) and not the 
condition number of an algorithm that may be used to evaluate P(z). 


Facts: 
For proofs and additional background, see, for example, [Dat95], [GV96], [Ste98], or [Wil65]. 


1. Because rounding errors are ubiquitous, a finite precision computational procedure can at best 
produce P(z + 6z) where, in a suitably chosen norm, ||5z|| < ¢||z|| and ¢ is a modest multiple of 
the unit round of the floating point system. (See section 37.6.) 

2. The relative condition number determines the tight, asymptotic relative error bound 


|P@ +52) — PI nd (a) lth 9 ( Wh) 
lzlI 


< co 
II P (z)I \|z| 
as 6z tends to zero. Very roughly speaking, if the larger components of the data z have p correct 
significant digits and the condition number is condp(z) ~* 10°, then the larger components of the 
result P(z) have p — s correct significant digits. 
3. [Hig96, p. 9] If P (x) has a Frechet derivative D(z) at z € F”, then the relative condition number is 


_ | D(@)Il llzll 
cond p(z) = TP@il 


In particular, if f(x) isa smooth real function ofa real variable x, then cond ¢(z) = |zf’(z)/f(z)|- 
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Examples: 
1. If P(x) = sin(x) and the nominal data point z = 22/7 may be in error by as much as 2 — 22/7 © 


.00126, then P(z) = sin(z) may be in error by as much as 100%. With such an uncertainty in 
Z = 22/7, sin(z) may be off by 100%, ie., sin(z) may have relative error equal to one. In most 
circumstances, z = 22/7 is considered to be ill-conditioned. 

The condition number of z € R with respect to sin(z) is cond,;,(z) = |zcot(z)|, and, in 
particular, cond(22/7) ~ 2485.47. If z = 22/7 is perturbed to z + 5z = a, then the asymptotic 
relative error bound in Fact 2 becomes 


sin(z + 6z) — sin(z) 


< cond(z) 


6 
| + o«iee)/20 
= 0.9999995...+ 0(|(6z)/z|). 


sin(z) 


sin(z+6z)—sin(z) 
sin(z) 


=1. 


The actual relative error in sin(z) is 


. Subtractive Cancellation: For x € RR’, define P(x) by P(x) = [1, —1]x. The gradient of P(x) is 


Vv P(x) = [1, —1] independent of x, so, using the oo-norm, Fact 3 gives 


IV FilloollXlloo _ 2max {|x1|, |x2]} 


IF@llo [x1 — x 


cond p(x) = 


Reflecting the trouble associated with subtractive cancellation, condp(x) shows that x is ill- 
conditioned when x; * x2. 
Conditioning of Matrix—Vector Multiplication: More generally, for a fixed matrix A € F”*" that is 
not subject to perturbation, define P(x) : F” — F” by P(x) = Ax. The relative condition number 
of x € F” is 
IIxIl 

cond(x) = || A|| jax (37.2) 
where the matrix norm is the operator norm induced by the chosen vector norm. If A is square 
and nonsingular, then cond(x) < || Al] || A7'|]. 
Conditioning of the Polynomial Zeros: Let q(x) = x? — 2x + 1 and consider the computational 
task of determining the roots of q(x) from the power basis coefficients [1, —2, 1]. Formally, the 
computational problem is to evaluate the function P : R*? — C that maps the power basis 
coefficients of quadratic polynomials to their roots. If q(x) is perturbed to q(x) + ¢, then the roots 
change from a double root at x = 1 tox = 1+ ,/e. A relative error of ¢ in the data [1, —2, 1] 
induces a relative error of ,/Jé| in the roots. In particular, the roots suffer an infinite rate of change 
at € = 0. The condition number of the coefficients [1, —2, 1] is infinite (with respect to root 
finding). 

The example illustrates the fact that the problem of calculating the roots of a polynomial q from 
its coefficients is highly ill-conditioned when q has multiple or near multiple roots. Although it is 
common to say that “multiple roots are ill-conditioned,” strictly speaking, this is incorrect. It is the 
coefficients that are ill-conditioned because they are the initial data for the calculation. 


[Dat95, p. 81], [Wil64, Wil65] Wilkinson Polynomial: Let w(x) be the degree 20 polynomial 
w(x) = (x — 1)(x — 2)... (x — 20) = x7 — 210x!? + 20615x!8 .. - + 2432902008176640000. 
The roots of w(x) are the integes 1, 2, 3, ... , 20. Although distinct, the roots are highly ill- 


conditioned functions of the power basis coefficients. For simplicity, consider only perturba- 
tions to the coefficient of x!°. Perturbing the coefficient of x!” from —210 to —210 — 277 
* 210 — 1.12 x 107’ drastically changes some of the roots. For example, the roots 16 and 17 
become a complex conjugate pair approximately equal to 16.73 + 2.811. 
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Let Pi6(z) be the root of w(x) = w(x) + (z— 210)x!? nearest 16 and let P,7(z) be the root nearest 
17. So, for z = 210, Pi6(z) = 16 and P,7(z) = 17. The condition numbers of z = 210 with 
respect to Pyg and Pz are cond)¢(210) = 210(16!°/(16w'(16))) * 3 x 10!° and cond,7(210) = 


210(17!°/(17w'(17))) © 2 x 101°, respectively. The condition numbers are so large that even per- 
I|5zl| 


turbations as small as 2~° are outside the asymptotic region in which 0 (1) is negligible in Fact 2. 


37.5 Conditioning of Linear Systems 


This section applies conditioning concepts to the computational task of finding a solution to the system 
of linear equations Ax = b for a given matrix A € R"*” and right-hand side vector b € R”. 

Throughout this section, A € R”*” is nonsingular. Let the matrix norm || - || be an operator matrix 
norm induced by the vector norm || - ||. Use || Al] + ||b]] to measure the magnitude of the data A and b. If 
E € R"*" isa perturbation of A andr € R” is a perturbation of b, then || E || + ||r|| is the magnitude of 
the perturbation to the linear system Ax = b. 


Definitions: 


The norm-wise condition number of a nonsingular matrix A (for solving a linear system) is k(A) = 
|| A71|| || All. If A is singular, then by convention, «(A) = oo. For a specific norm || - ||,,, the condition 
number of A is denoted k,,(A). 


Facts: 
For proofs and additional background, see, for example, [Dat95], [GV96], [Ste98], or [Wil65]. 


1. Properties of the Condition Number: 
(a) K(A) > 1. 
(b) K(AB) < K(A)x(B). 
(c) k(a@A) = k(A), for all scalars a # 0. 


(d) «2(A) = 1lifand only if Aisa nonzero scalar multiple of an orthogonal matrix, i.e. A7A = al 
for some scalar a. 


(e) k2(A) = k(A‘). 
(f) K2(ATA) = («2(A))?. 


(g) K2(A) = |[All2l] AT le = Omax/Omins Where Omax aNd Omin are the largest and smallest singular 
values of A. 
2. For the p-norms (including || - ||1, || - ll2, and || - lleoo), 


1 . | All 
= min 
k(A) |All 
So, «(A) is one over the relative-to-|| A|| distance from A to the nearest singular matrix, and, in 
particular, «(A) is large if and only if a small-relative-to-|| A|| perturbation of A is singular. 


3. Regarding A as fixed and not subject to errors, it follows from Equation 37.2 that the condition 
number of b with respect to solving Ax = b as defined in Equation 37.1 is 


A+ 6dAis sigan ; 


_ | ATtT Mb 


cond(b) = ]A"b] < «(A). 


If the matrix norm is || A~'|| is induced by the vector norm ||b||, then equality is possible. 
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4. Regarding b as fixed and not subject to errors, the condition number of A with respect to solving 
Ax = bas defined in Equation 37.1 is cond(A) = ||A7'|| || Al] = «(A). 


5. K(A) < cond([A, b]) < (an) «(A), where cond([A, b]) is the condition number of the 


data [A, b] with respect to solving Ax = b as defined in Equation 37.1. Hence, the data [ A, b] are 
norm-wise ill-conditioned for the problem of solving Ax = b if and only if « (A) is large. 

6. Ifr = b — A(x + 6x), then the 2-norm and Frobenius norm smallest perturbation 6A € R"*" 
satisfying (A + 6A)(x + 8x) =bisd6A = % and |S All. = |[SAl|p = EL 


~~ xt x IIxll2° 


7. Let 5A and db be perturbations of the data A and b, respectively. If || A~'d.Al| < 1, then A+65A is 
nonsingular, there is a unique solution x + 5x to (A + 6A)(x + 6x) = (b— 6b), and 


-1 
loxll — — WAIAT Goeacee 


xl] ~ (2 —|[ATTSAI]) \ WAI || bj] 
Examples: 
1. An Ill-Conditioned Linear System: For € € R, let A = 1 : , and b = ] . Fore 4 0, Ais 


nonsingular and x = | satisfies Ax = b. The system of equations is ill-conditioned when ¢ is 
small because some small changes in the data cause a large change in the solution. For example, 


perturbing b to b+ db, where 5b = 


; Oo}. 
€ R’, changes the solution x tox+ 6x = | independent 


of the choice of ¢ no matter how small. 

Using the 1-norm, «;(A) = ||A7!||: || All: = (2 + €)e7!. As € tends to zero, the perturbation 
5b tends to zero, but the condition number x) (A) explodes to infinity. 

Geometrically, x is gives the coordinates of the intersection of the two lines x + y = 1 and 
x + (1+ e)y = 1. Ife is small, then these lines are nearly parallel, so a small change in them may 
move the intersection a long distance. 


: : Fae fe cate 6 . 
Also notice that the singular matrix | 1 i is a € perturbation of A. 


2. A Well-Conditioned Linear System Problem: Let A = | . For b € R?, the solution to Ax = b 
‘ 1 |b, +b2 : ‘ ! 
isk= 5], by” In particular, perturbing b to b + 5b changes x to x + 45x with ||6x||; < ||b||; 
1— 92 


and ||6x||2 = ||db]l2, i-e., x is perturbed by no more than b is perturbed. This is a well-conditioned 
system of equations. 

The l-norm condition number of A is k;(A) = 2, and the 2-norm condition number is 
K2(A) = 1, which is as small as possible. 

Geometrically, x gives the coordinates of the intersection of the perpendicular lines x + y = 1 
and x — y = 1. Slighly perturbing the lines only slightly perturbs their intersection. 


Also notice that for both the 1-norm and 2-norm an || Ax|| = 1, so no small-relative-to-|| A|| 
x|/=1 


perturbation of A is singular. If A + 5A is singular, then ||5 A|| > 1. 
3. Some Well-known Ill-conditioned Matrices: 


(a) The upper triangular matrices B, € R” of the form 
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have oo-norm condition number k. = n2"~!, Replacing the (n, 1) entry by —2?-” makes B, 
singular. Note that the determinant det(B,,) = 1 gives no indication of how nearly singular the 
matrices B,, are. 


(b) The Hilbert matrix: The order n Hilbert matrix H,, € R"*” is defined by hjj = 1/(i + j — 1). 
The Hilbert matrix arises naturally in calculating best Ly polynomial approximations. The 
following table lists the 2-norm condition numbers to the nearest power of 10 of selected 
Hilbert matrices. 


| n |] 1] 2 3 4 5 6 7 8 9 10 
| «2(Hn): | 1 | 10 | 10° | 10* | 10° | 10” | 10® | 10° | 10!! | 10” 


(c) Vandermonde matrix: The Vandermonde matrix corresponding tox € R" is Vx € R"*" given by 
vj; = x; ’. Vandermonde matrices arise naturally in polynomial interpolation computations. 
The following table lists the 2-norm condition numbers to the nearest power of 10 of selected 
Vandermonde matrices. 


| n |] 1] 2 | 3 4 5 6 7 8 9 10 
[ «2 (Vi23....mj): | 1 | 10 | 10 | 10% | 10% | 10> | 107 | 10° | 10 | 10” 


37.6 Floating Point Numbers 


Most scientific and engineering computations rely on floating point arithmetic. At this writing, the 
IEEE 754 standard of binary floating point arithmetic [IEEE754] and the IEEE 854 standard of radix- 
independent floating point arithmetic [IEEE854] are the most widely accepted standards for floating 
point arithmetic. The still incomplete revised floating point arithmetic standard [[EEE754r] is planned 
to incorporate both [[EEE754] and [IEEE854] along with extensions, revisions, and clarifications. See 
[Ove01] for a textbook introduction to IEEE standard floating point arithmetic. 

Even 20 years after publication of [IEEE754], implementations of floating point arithmetic vary in so 
many different ways that few axiomatic statements hold for all of them. Reflecting this unfortunate state of 
affairs, the summary of floating point arithmetic here is based upon IEEE 754r draft standard [IEEE754r] 
(necessarily omitting most of it), with frequent digressions to nonstandard floating point arithmetic. 

In this section, the phrase standard-conforming refers to the October 20, 2005 IEEE 754r draft standard. 


Definitions: 


A p-digit, radix b floating point number with exponent bounds eyax and emin is a real number of the 
form x = + ( pet) b°, where e is an integer exponent, @min < € < @max, and m is a p—digit, base b 
integer significand. The related quantity m/b? is called the mantissa. Virtually all floating point systems 
allow m = 0 and b?-! < m < b?. Standard-conforming, floating point systems allow all significands 
0 < m < D?. If two or more different choices of significand m and exponent e yield the same floating 
point number, then the largest possible significand m with smallest possible exponent e is preferred. 

In addition to finite floating point numbers, standard-conforming, floating point systems include 
elements that are not numbers, including 00, —oo, and not-a-number elements collectively called NaNs. 
Invalid or indeterminate arithmetic operations like 0/0 or oo — 00 as well as arithmetic operations involving 
NaNs result in NaNs. 

The representation +(m/b?~!)b® of a floating point number is said to be normalized or normal, if 
bP! <m<b?. 

Floating point numbers of magnitude less than b*™® are said to be subnormal, because they are too small 
to be normalized. The term gradual underflow refers to the use of subnormal floating point numbers. 
Standard-conforming, floating point arithmetic allows gradual underflow. 
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For x € R, a rounding mode maps x to a floating point number fl(x). Except in cases of overflow 
discussed below, fl(x) is either the smallest floating point number greater than or equal to x or the 
largest floating point number less than or equal to x. Standard-conforming, floating point arithmetic 
allows program control over which choice is used. The default rounding mode in standard conforming 
arithmetic is round-to-nearest, ties-to-even in which, except for overflow (described below), fl(x) is the 
nearest floating point number to x. In case there are two floating point numbers equally distant from x, 
fl(x) is the one with even significand. 

Underflow occurs in fl(x) = 0 when 0 < |x| < b°™". Often, underflows are set quietly to zero. Gradual 
underflow occurs when fl(x) is a subnormal floating point number. Overflow occurs when |x| equals or 
exceeds a threshold at or near the largest floating point number (b — b'~?)b*»™, Standard-conforming 
arithmetic allows some, very limited program control over the overflow and underflow threshold, whether 
to set overflows to oo and whether to trap program execution on overflow or underflow in order to take 
corrective action or to issue error messages. In the default round-to-nearest, ties-to-even rounding mode, 
overflow occurs if |x| > (b — sbi P bens, and in that case, fl(x) = too with the sign chosen to agree 
with the sign of x. By default, program execution continues without traps or interruption. 

A variety of terms describe the precision with which a floating point system models real numbers. 


* The precision is the number p of base-b digits in the significand. 

* Big M is the largest integer M with the property that all integers 1, 2, 3,... , M are floating point 
numbers, but M + 1 is not a floating point number. If the exponent upper bound emax is greater 
than the precision p, then M = DP. 

* Themachineepsilon, ¢ = b!~?, is the distance between the number one and the next larger floating 
point number. 

° The unit round u = inf{é > 0| fl(1 +6) > 1}. Depending on the rounding mode, u may be as 
large as the machine epsilon €. In round-to-nearest, ties-to-even rounding mode, u = 3: 


In standard-conforming, floating point arithmetic, if and £ are floating point numbers, then floating 
point addition ©, floating point subtraction ©, floating point multiplication ©, and floating point 
division @ are defined by 


a ® B=fl(a +B), (37.3) 
a © B =fl(a — B), (37.4) 
a@Bp=fl(a x B), (37.5) 
a@p=fl(a + B), (37.6) 


The IEEE 754r [IEEE754r] standard also includes a fused addition-multiply operation that evaluates wB+ y 
with only one rounding error. 

In particular, if the exact, infinitely precise value of a + B, a — B,a x B, or a ~ B is also a floating 
point number, then the corresponding floating point arithmetic operation occurs without rounding error. 
Floating point sums, products, and differences of small integers have zero rounding error. 

Nonstandard-conforming, floating point arithmetics do not always conform to this definition, but often 
they do. Even when they deviate, it is nearly always the case that if e is one of the arithmetic operations 
+, —, x, or + and © is the corresponding nonstandard floating point operation, then a © f isa floating 
point number satisfying a © 6 = a(1 + da) e B(1 + 5B) with |5a| < b?-? and |6p| < b?-?. 

If e is one of the arithmetic operations +, —, x, or + and © is the corresponding floating point 
operation, then the rounding error in a © B is (a e 8) — (a - B), i.e., rounding error is the difference 
between the exact, infinitely precise arithmetic operation and the floating point arithmetic operation. In 
more extensive calculations, rounding error refers to the cumulative effect of the rounding errors in the 
individual floating point operations. 

In machine computation, truncation error refers to the error made by replacing an infinite process by 
a finite process, e.g., truncating an infinite series of numbers to a finite partial sum. 
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Many computers implement floating point numbers of two or more different precisions. Typical single 
precision floating point numbers have machine epsilon roughly 10~’ and precision roughly 7 decimal 
digits or 24 binary digits. Typical double precision floating point numbers have machine epsilon roughly 
10—'¢ and precision roughly 16 decimal digits or 53 binary digits. Specification of IEEE standard arithmetic 
[IEEE754r] includes these three precisions. See the table in Example 1. In addition, it is not unusual to 
also implement extended precision floating point numbers with even greater precision. 

If X € F is an approximation to x ¢€ F, the absolute error in % is |x — X| and the relative error in x is 
|(x — &)/x|. If x = 0, then the relative error is undefined. 

Subtractive cancellation of significant digits occurs in floating point sums when the relative error 
in the rounding-error-corrupted approximate sum is substantially greater than the relative error in the 
summands. In cases of subtractive cancellation, the sum has magnitude substantially smaller than the 
magnitude of the individual summands. 


Facts: 


For proofs and additional background, see, for example, [Ove01]. 
In this section, we make the following assumptions. 


* The numbers a, 6, and y are p-digit radix b floating point numbers with exponent bounds emax 
and €min- 

* The floating point arithmetic operations satisfy Equation 37.3 to Equation 37.6. 

* In the absence of overflows, the rounding mode fl(x) maps x € R to the closest floating point 
number or to one of the two closest in case of a tie. In particular, the unit round is 3b : 


Standard-conforming arithmetic in round-to-nearest, ties-to-even rounding mode satisfies these 
assumptions. 

For vectors x € R” and matrices M € R™*", the notation |x| and |M| indicates the vector and matrix 
whose entries are the absolute values of the corresponding entries of x and M, respectively. For x, y € R” 
the inequality x < y represents the n scalar inequalities x; < yj,i = 1, 2, 3,..., n. Similarly, for 
A, B € R”*", the inequality A < B represents the mn inequalities aj; < bjj,i = 1, 2,3,..., mand 
j =1,2,3,...,0. 

If e is an arithmetic expression involving only floating point numbers, the notation fl(e) represents the 


ij 


expression obtained by replacing each arithmetic operation by the corresponding floating point arithmetic 
operation 37.3, 37.4, 37.5, or 37.6. Note that fl(-) is not a function, because its value may depend on the 
order of operations in e—not value of e. 


1. At this writing, the only radixes in common use on computers and calculators are b = 2andb = 10. 

2. The commutative properties of addition and multiplication hold for floating point addition 37.3 
and floating point multiplication 37.5, ie,a © B = B @a anda @ B = B @a. (Examples 
below show that the associative and distributive properties of addition and multiplication do not 
in general hold for floating point arithmetic.) 

3. In the absence of overflow or any kind of underflow, fl(x) = x(1 + 4) with || < 5b. 

4. Rounding in Arithmetic Operations: If e is an arithmetic operation and © is the corresponding 
floating point operation, then 


a © B= (ae B)(1 +54) 


with |3| < 1b!-?. The error 5 may depend on @ and £ as well as the arithmetic operation. 

5. Differences of Nearby Floating Point Numbers: If a > 0 and B > 0 are normalized floating point 
numbers and ; <a/B < 2, then fl(a — 8B) =a 6 B =a — B,e., there is zero rounding error in 
floating point subtraction of nearby numbers. 
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6. Product of Floating Point Numbers: Ifa;,i = 1, 2,3,... n aren floating point numbers, then 


fl (I1«:) = (I1«.) (1+ 6)" 


with |5| < A . Consequently, rounding errors in floating point products create only minute 
relative errors. 

7. [Ste98], [Wil65] Dot (or Inner) Product of Floating Point Numbers: Let x € R" andy € R” be 
vectors of floating point numbers and let s be the finite precision dot product s = fl(x’y) = 
X1 @ Vy, BX @ yo O X3 @ y3 +++ PB Xy ® Yy evaluated in the order shown, multiplications first 
followed by additions from left to right. Ifn < 0.1/ (5b? ) and neither overflow nor any kind of 
underflow occurs, then the following hold. 


(a) s = fi(x’y) = xiyi(1 + 81) + x2y2(1 + 52) + x3y3(1 + 83) + +++ + Xn¥n(1 + 63) with 

\5;| < 1.06n(5b'-?) and |5;| < 1.06(n — j + 2)(5b'-?) < 1.06n(5b'-?), for j = 2, 3,4, 
.yn. 

(b) s= fl(xTy) = xy for some vectors X, ¥ € R" satisfying |x — X| < |x|(1+ 1.06n(5b'-?)) and 


ly—-yl<lyldt+ 1.06n(5b'~?)). So, s is the mathematically correct product of the vectors x 
and ¥ each of whose entries differ from the corresponding entries of x or y by minute relative 
errors. 


There are infinitely many choices of X and y. In the notation of Fact 7(a), two are £; = 
xj(1+6;), pj = yj and&; = x;(14+ 6;)', 9; = yj) +4;)'7, j = 1,2,3,...50. 
(c) Ifx?y ¢ 0, then the relative error in s is bounded as 


s—xly 
xTy 


Ix|7 Ly| 
|xTy| 


< 1.06n(b'-?) 


The bound shows that if there is little cancellation in the sum, then s has small relative error. The 
bound allows the possibility that s has large relative error when there is substantial cancellation 
in the sum. Indeed this is often the case. 


8. Rounding Error Bounds for Floating Point Matrix Operations: In the following, A and B are matrices 
each of whose entries is a floating point number, x is a vector of floating point numbers, and c 
is a floating point number. The matrices A and B are compatible for matrix addition or matrix 
multiplication as necessary and E is an error matrix (usually different in each case) whose entries 
may or may not be floating point numbers. The integer dimension n is assumed to satisfy n < 
0.1/(5b'?). 


If neither overflows nor any kind of underflow occurs, then 

(a) fl(cA) =cA+ E with |E| < 3b' PleAl. 

(b) fl(A+ B) = (A+ B)+ E with |E| < shh PlA+ B\. 

(c) Ifx € R” and A € R™*", then fl(Ax) = (A+ E)x with |E| < 1.06n(5b'-?). 

(d) Matrix multiplication: If A € R”*" and B € R"*4,fl(AB) = AB+E with|E| < 1.06n(5b'-?) 
|A||BI. 
Note that if |AB| ~ |A||B|, then each entry in fl(AB) is correct to within a minute relative 


error. Otherwise, subtractive cancellation is possible and some entries of fl(A B) may have large 
relative errors. 


(e) Let || - || be a matrix norm satisfying || E || < || |E| ||. (All of || - Il I+ lls Il + Ips I+ Iloo and 
|| - |lp satisfy this requirement.) If A € R”*” and B € R"”4, then fl(AB) = AB + E with 
[Ell < 1.06n(5b'-?)I| |AJ |] I] | BI I. 
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(f) Matrix multiplication by an orthogonal matrix: If Q € R"*" is an orthogonal matrix, ie., if 
Q’ = Q”| and if A € R™™", then 


(i) f(QA) = QA+E with||E ||. < 1.06n°(56'-P) || Ally and [IE llr < 1.06? (46'-?) | All. 


(ii) f(QA) = Q(A+E) with ||Ell. < 1.061? (!-P) | All> and [LE llr < 1.06n9 (F6'-P) | Alle. 


Note that this bound shows that if one of the factors is an orthogonal matrix, then subtractive 
cancellation in floating point matrix multiplication is limited to those entries (if any) that 
have magnitude substantially smaller than the other, possibly nonorthogonal factor. Many 
particularly successful numerical methods derive their robustness in the presence of rounding 
errors from this observation. (See, for example, [Dat95], [GV96], [Ste98], or [Wil65].) 


Examples: 


1. The following table lists typical floating point systems in common use at the time of this writing. 


radixb precision p Pini Siaak 
Some calculators 10 14 -99 99 
Some calculators 10 14 -999 999 
IEEE 754r decimal 64 [I[EEE754r] 10 16 -383 384 
IEEE 754r decimal 128 [IEEE754r] 10 34 -6143 6144 
IEEE 754r binary 32 (Single) [IEEE754r] 2 24 -126 127 
IEEE 754r binary 64 (Double) [[EEE754r] 2 53 -1022 1023 
IEEE 754r binary 128 (Double Extended) [IEEE754r] 2 113. -16382 = 16383 


2. Consider p = 5 digit, radix b = 10 floating-point arithmetic with round-to-nearest, ties-to-even 
rounding mode. 


(a) 


(b) 


(c) 


Floating point addition is not associative: Ifa = 1, 8 = 10°,and y = —10°, then (a@ 8) By = 0 
but a @ (6 @ y) = 1. (This is also an example of subtractive cancellation.) 
Floating point multiplication/division is not associative: If a = 3, B = 1, and y = 3, then 


a ®(B@y) = .99999 but (a ® B) @y = 1. Ifa = 44444, B = 55555 and y = 66666, then 
a ® (B ® y) = 1.6460 x 10!* but (a @ B) @ y = 1.6461 x 10!*. Although different, both 
expressions have minute relative error. It is generally the case that floating point products have 
small relative errors. (See Fact 6.) 


Floating point multiplication does not distribute across floating point addition: Ifa = 9, B = 1, and 
y = —.99999, then a @ (6 @ y) = 9.0000 x 10~° but (a @ B) @ (a @ y) = 1.0000 x 107+. 


3. Subtractive Cancellation: In p = 5, radix b = 10 arithmetic with round-to-nearest, ties-to-even 
rounding, the expression (1+ 10-4 — 1)/10~* evaluates as follows. (Here we assume that the 
floating point evaluation of the square root gives the same result as rounding the exact square root to 
p =5, radix b = 10 digits.) 


(fl(,/1.0000 @ 10-4) 6 1) @ 1074 = (fl(1.0001) 6 1.0000) @ 1074 
= (f(1.000049999 ...) © 1.0000) @ 1074 
= (1.0000 © 1.0000) @ 1074 
= 0.0000. 


In exact, infinite precision arithmetic, (/1 + 10-4 — 1)/10~* = .4999875..., so the relative error is 
1. Note that zero rounding error occurs in the subtraction. The only nonzero rounding error is the 
square root, which has minute relative error roughly 5 x 10~°, but this is enough to ruin the final 
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result. Subtractive cancellation did not cause the large relative error. It only exposed the unfortunate 
fact that the result was ruined by earlier rounding errors. 

4. Relative vs. Absolute Errors: Consider &; = 31416 as an approximation to x; = 10000 and %, = 3.07 
as an approximation to x. = z, The absolute errors are nearly the same: |X, — x,| * —0.0735 and 


: : ki — x . 
|X — x2| © —0.0716. On the other hand, the relative error in the first case, ae Al ~2~x 10~°, is 
. Ix;| 
: ; |X. — X2| 2g ; 
much smaller than the relative error in the second case, Ta x 2 x 107°. The smaller relative 
x2 
error shows that X; = 31416 is a better approximation to 10000z than x) = 3.07 is to z. The absolute 


errors gave no indication of this. 


37.7. Algorithms and Efficiency 


In this section, we introduce efficiency of algorithms. 


Definitions: 


Analgorithm isa precise set of instructions to perform a task. The algorithms discussed here perform math- 
ematical tasks that transform an initial data set called the input into a desired result final data set called the 
output using an ordered list of arithmetic operations, comparisons, and decisions. For example, the Gaus- 
sian elimination algorithm (see Chapter 39.3) solves the linear system of equations Ax = b by transforming 
the given, nonsingular matrix A € R”*" and right-hand-side b € R” into a vector x satisfying Ax = b. 

One algorithm is more efficient than another if it accomplishes the same task with a lower cost of 
computation. Cost of computation is usually dominated by the direct and indirection cost of execution 
time. So, in general, in a given computational environment, the more efficient algorithm finishes its task 
sooner. (The very real economic cost of algorithm development and implementation is not a part of the 
efficiency of an algorithm.) However, the amount of primary and secondary memory required by an 
algorithm or the expense of and availability of the necessary equipment to execute the algorithm may also 
be significant part of the cost. In this sense, an algorithm that can accomplish its task on an inexpensive, 
programmable calculator is more efficient than one that needs a supercomputer. 

For this discussion, a floating point operation or flop consists of a floating point addition, subtraction, 
multiplication, division, or square root along with any necessary subscripting and loop index overhead. In 
FortranA(I,J) = A(I,J) + C*A(K,J) performs two flops. (Note that this is a slightly different 
definition of flop than is used in computer engineering.) 

Formal algorithms are often specified in terms of an informal computer program called pseudo-code. 

On early digital computers, computation time was heavily dominated by evaluating floating point 
operations. So, traditionally, numerical analysts compare the efficiency of two algorithms by counting the 
number of floating point operations each of them executes. If n measures the input data set size, e.g., an 
input matrix A € R"*”, then an O(n?) algorithm is one that, for some positive constant c, performs cn? 
plus a sum of lower powers of n floating point operations. 


Facts: 


For proofs and additional background, see, for example, [Dat95], [GV96], [Ste98], or [Wil65]. 

In choosing a numerical method, efficiency must be balanced against considerations like robustness 
against rounding error and likelyhood of failure. 

Despite tradition, execution time has never been more than roughly proportional to the amount of 
floating point arithmetic. On modern computers with fast floating point arithmetic, multiple levels of 
cache memory, overlapped instruction execution, and parallel processing, execution time is correlated 
more closely with the number of cache misses (i.e., references to main RAM memory) than it is to the 
number of floating point operations. In addition, the relative execution time of algorithms depends strongly 
on the environment in which they are executed. 
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Nevertheless, for algorithms highly dominated by floating point arithmetic, flop counts are still useful. 
Despite the complexities of modern computers, flop counts typically expose the rate at which execu- 
tion time increases as the size of the problem increases. Solving linear equations Ax = b for given 


A € R™" and b € R" by Gaussian elimination with partial pivoting is an O(n*) algorithm. For larger 
values of n, solving a 2n-by-2n system of equations takes roughly eight times longer than an n-by-n 
system. 


1. 


Triangular back substitution is an O(n’) algorithm to solve a triangular system of linear equations 
Tx = b,b € R" and T € R"*", with #;; = 0 whenever i > j [GV96]. (See the pseudo-code 
algorithm below.) 

[GV96] Gaussian elimination with partial pivoting (cf. Algorithm 1, Section 38.3) is an O(n?) 

algorithm to solve a system of equations Ax = b with A € R”*" andb € R”. 

Because of the need to repeatedly search an entire submatrix, Gaussian elimination with complete 

pivoting is an O(n*) algorithm to solve a system of equations Ax = b with A € R’™" and b € R" 

[GV96]. Hence, complete pivoting is not competitive with O(n?) methods like Gaussian elimination 

with partial pivoting. 

[GV96] The QR-factorization by Householder’s method is an O(n) algorithm to solve a system 
of equations Ax = b with A € R"*" andb € R". 

[GV96] The QR factorization by Householder’s method to solve the least squares problem min || Ax— 
b||2 for given A € R”*" and b € R” is an O(n?(m — n/3)) algorithm. 

[GV96] The singular value decomposition using the Golub—Kahan-Reinsch algorithm to solve the 

least squares problem min || Ax — b|| for given A € R™*" and b € R” is an O(m?n + mn? + n°) 

algorithm. 

[GV96] The implicit, double-shift QR iteration algorithm to find all eigenvalues of a given matrix 

A € R™" is an O(n?) algorithm. 

Cramer’s rule for solving the system of equations Ax = b for given A € R”*” and b € R" in 

which determinants are evaluated using minors and cofactors is an O((” + 1)n!) algorithm and is 

impractical for all but small values of n. 
Cramer’s rule for solving the system of equations Ax = b for given A € R”’*” and b € R" in 
which determinants are evaluated using Gaussian elimination is an O(n‘) algorithm and is not 


competitive with O(n?) methods like Gaussian elimination with partial pivoting. 


Examples: 


1. 


It takes roughly 4.6 seconds on a 2GHz Pentium workstation, using Gaussian elimination with 
partial pivoting, to solve the n = 2000 linear equations in 2000 unknowns Ax = b in which the 
entries of A and b are normally distributed pseudo-random numbers with mean zero and variance 
one. It takes roughly 34 seconds to solve a similar n = 4000 system of equations. This is consistent 
with the estimate that Gaussian elimination with partial pivoting is an O(n’) algorithm. 


. This is an example of a formal algorithm specified in pseudo-code. Consider the problem of solving 


for y in the upper triangular system of equations Ty = b, where b € R” is a given right-hand-side 
vector and T € R"*" is a given nonsingular upper triangular matrix; ie., 4; = 0 fori > j and 
ti; #4 Ofori = 1,2,3,...,n. 

Input: A nonsingular, upper triangular matrix T € R"*” and a vector b € R”. 

Output: The vector y € R” satisfying Ty = b. 

Step 1. yn < Un/ tan 

Step 2. Fori = n—1,n—2,...,2,1 do 


n 


2.1 5s; <b; — x GiVj 
jit 
22 yi= Sj / tii 
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37.8 Numerical Stability and Instability 


Numerically stable algorithms, despite rounding and truncation errors, produce results that are roughly 
as accurate as the errors in the input data allow. Numerically unstable algorithms allow rounding and 
truncation errors to produce results that are substantially less accurate than the errors in the input data 
allow. This section concerns numerical stability and instability and is loosely based on [Bun87]. 


Definitions: 
A computational problem is the task of evaluating a function f : R” — R” at a particular data point 
x € R". A numerical algorithm that is subject to rounding and truncation errors evaluates a perturbed 
function f(x). Throughout this section ¢ represents a modest multiple of the unit round. 

The forward error is f (x) — f (x), the difference between the mathematically exact function evaluation 
and the perturbed function evaluation. 

The backward error is a vector e € R" of smallest norm for which f(x+e) = F(x). If no such e exists, 
then the backward error is undefined. This definition is illustrated in Figure 37.1 [Ste98, p. 123]. 

An algorithm is forward stable if, despite rounding and truncation errors, 


If@) - Fl — , 
If@l 


for all valid input data x. In a forward stable algorithm, the forward relative error is small for all valid input 
data x despite rounding and truncation errors in the algorithm. 

An algorithm is backward stable or strongly stable if the backward relative error e exists and satisfies 
the relative error bound |le|| < ¢||x|| for all valid input data x despite rounding and truncation errors. 

In this context, “small” means a modest multiple of the size of the errors in the data x. If rounding errors 
are the only relevant errors, then “small” means a modest multiple of the unit round. 


Domain Range 


JO19 pseMio4 


Backward 
Error 


f(x) 


FIGURE 37.1 A computational problem is the task of evaluating a function f at a particular data point x. A 
numerical algorithm that is subject to rounding and truncation errors evaluates a perturbed function f (x). The 
forward error is f(x) — f(x), the difference between the mathematically exact function evaluation and the 
perturbed function evaluation. The backwarderror isa vectore € IR” ofsmallest norm for which f(x+e) = Ff (x) 
[Ste98, p. 123]. If no such vector e exists, then there is no backward error. 
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A finite precision numerical algorithm is weakly numerically stable if rounding and truncation errors 
cause it to evaluate a perturbed function f(x) satisfying the relative error bound 


f(x) — fll 
fx 


for all valid input data x. Ina weakly stable algorithm, the magnitude of the forward error is no greater than 
magnitude of an error that could be induced by perturbing the data by small multiple of the unit round. 
Note that this does not imply that there is a small backward error or even that a backward error exists. (An 
even weaker kind of “weak stability” requires the relative error bound only when x is well-conditioned 
[Bun87].) 

An algorithm is numerically stable if rounding errors and truncation errors cause it to evaluate a 
perturbed function f(x) satisfying 


< econd (x) 


If&+e)— fll — , 
fC 


for some small relative-to- ||x|| backward error e, |le|| < ¢||x||. Ina numerically stable algorithm, f(x) lies 
near a function with small backward error. 

Figure 37.2 illustrates the definitions of stability. The black dot on the left represents a nominal data point 
x. The black dot on the right is the exact, unperturbed value of f (x). The shaded region on the left represents 
the small relative-to-||x|| perturbations of x. The shaded region on the right is its exact image under f (x). 

Ina weakly stable numerical method, the computed function value f (x) lies inside the large circle with 
radius equal to the longest distance from the black dot f (x) to the furthest point in the shaded region con- 
taining it. The error ina weakly stable algorithm is no larger than would have been obtained from a backward 
stable algorithm. However, the actual result may or may not correspond to a small perturbation of the data. 

In a numerically stable algorithm, f(x) lies either near or inside the shaded region on the right. 

In a backward stable algorithm, f (x) lies in the shaded region, but, if the data are ill-conditioned as 
in the illustration, f(x) may have a large relative error. (To avoid clutter, there is no arrow illustrating a 
backward stable algorithm.) 

In a forward stable algorithm, f(x) has a small relative error, but f (x) may or may not correspond to 
a small perturbation of the data. 


Forward Stable 


Numerically Stable 


Small 
Relative 
Backward 
Errors 


Weakly Numerically Stable 


FIGURE 37.2. The black dot on the left represents a nominal data point x. The black dot on the right is the exact, 
unperturbed value of f(x). The shaded region on the left represents the small relative-to-||x|| perturbations of 
x. The shaded region on the right is its exact image under f(x). The diagram illustrates the error behavior of a 
weakly numerically stable algorithm, a numerically stable algorithm, and a forward stable algorithm. 
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Facts: 


1. A numerically stable algorithm applied to an ill-conditioned problem may produce inaccurate 
results. For ill-conditioned problems even small errors in the input data may lead to large errors 
in the computed solution. Just as no numerical algorithm can reliably correct errors in the data 
or create information not originally implicit in the data, nor can a numerical algorithm reliably 
calculate accurate solutions to ill-conditioned problems. 

2. Rounding and truncation errors in a backward stable algorithm are equivalent to a further pertur- 
bation of the data. The computed results of a backward stable algorithm are realistic in the sense that 
they are what would have been obtained in exact arithmetic from an extra rounding-error-small 
relative perturbation of the data. Typically this extra error is negligible compared to other errors 
already present in the data. 

3. The forward error that occurs in a backward stable algorithm obeys the asymptotic condition 
number bound in Fact 2 of Section 37.4. Backward error analysis is based on this observation. 

4, [Dat95], [GV96] Some Well-Known Backward Stable Algorithms: 


(a) Fact 4 in Section 37.6 implies that a single floating point operation is both forward and backward 
stable. 


(b) Fact 7b in section 37.6 shows that the given naive dot product algorithm is backward stable. 
The algorithm is not, in general, forward stable because there may be cancellation of significant 
digits in the summation. 


(c) Gaussian elimination: Gaussian elimination with complete pivoting is backward stable. Gaus- 
sian elimination with partial pivoting is not, strictly speaking, backward stable. However, linear 
equations for which the algorithm exhibits instability are so extraordinarily rare that the algo- 
rithm is said to be “backward stable in practice” [Dat95, GV96]. 


(d) Triangular back substitution: The back-substitution algorithm in Example 2 in Section 37.7 
is backward stable. It can be shown that the rounding error corrupted computed solution x 
satisfies (T + E)& = b, where | ej; |< € | ti |, 1,7 = 1, 2, 3,..., . Thus, the computed 
solution X solves a nearby system. The back-substitution process is, therefore, backward stable. 


(e) QR factorization: The Householder and Givens methods for factorization of A = QR, where 
Q is orthogonal and R is upper triangular, are backward stable. 


(f) SVD computation: The Golub—Kahan-Reinsch algorithm is a backward stable algorithm for 
finding the singular value decomposition A = ULV", where © is diagonal and U and V are 
orthogonal. 


(g) Least-square problem: The Householder QR factorization, the Givens QR factorization, and 
Singular Value Decomposition (SVD) methods for solving linear least squares problems are 
backward stable. 


(h) Eigenvalue computations: The implicit double-shift QR iteration is backward stable. 


Examples: 


1. Anexample ofan algorithm that is forward stable but not backward stable is the natural computation 
of the outer product A = xy? from vectors x,y € R": fori, j = 1, 2,3,... n, set aij <— xj ® yj. 
This algorithm produces the correctly rounded value of the exact outer product, so it is forward 
stable. However, in general, rounding errors perturb the rank 1 matrix xy’ into a matrix of higher 
rank. So, the rounding error perturbed outer product is not equal to the outer product of any pair 
of vectors; i.e., there is no backward error, so the algorithm is not backward stable. 

2. Backward vs. Forward Errors: Consider the problem of evaluating f(x) = e* at x = 1. One 
numerical method is to sum several terms of the Taylor series for e*. If f(x) is approximated by 

2 3 
the truncated Taylor series f (x) =1l+x+ = + a then f (1) = e © 2.7183 and Ff) = 2.6667. 
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The forward error is f (1) — FQ) & 2.7183 — 2.6667 = 0.0516. The backward error is 1 — y, where 
f(Y= FQ), ie., the backward error is 1 — In( f (1) x 0192. 

3. A Numerically Unstable Algorithm: Consider the computational problem of evaluating the function 
f(x) = In(1 + x) for x near zero. The naive approach is to use fl(In(1 @ x)), i-e., add one to x in 
finite precision arithmetic and evaluate the natural logarithm of the result also in finite precision 
arithmetic. (For this discussion, assume that if z is a floating point number, then In(z) returns 
the correctly rounded exact value of In(z).) Applying this very simple algorithm to x = 107!° 
in p = 16, radix b = 10 arithmetic, gives fl(In(1 @ 107!°)) = fl(In(1)) = 0. The exact value 
of In(1 + 107!°) & 107!°, so the rounding error corrupted result has relative error 1. It is 100% 
incorrect. 

However, the function f(x) = In(1 + x) is well-conditioned when x is near zero. Moreover, 
lim,_,9 cond ;(x) = 1. The large relative error is not due to ill-conditioning. It demonstrates that 
this simple algorithm is numerically unstable for x near zero. 

4. Analternative algorithm to evaluate f(x) = In(1+ x) that does not suffer the above gross numerical 
instability for x near zero is to sum several terms of the Taylor series 


2 3 


x 
In(1 4 = t 
n( x)=x ; 5 


Although it is adequate for many purposes, this method can be improved. Note also that the series 
does not converge for |x| > 1 and converges slowly if |x| ~ 1, so some other method (perhaps 
fl(in(1 @ x))) is needed when x is not near zero. 

5. Gaussian Elimination without Pivoting: Gaussian elimination without pivoting is not numerically 
stable. For example, consider solving a system of two equations in two unknowns 


107 x34 2=1 
Xj +2x2=3 


using p = 9 digit, radix b = 10 arithmetic. Eliminating x, from the second equation, we obtain 


1071%x4+ = 1 
(2 6 10")x,=3 © 10!°, 


which becomes 


107! x, + Q= 1 
—10!x,=—10!°, 


giving x) = 1, x; = 0. The exact solution is x; = (1 — 2 x 107!) © 1, x) = (1-3 x 107!)/1— 
2x 10719) & 1, 


The oo-norm condition number of the coefficient matrix A = i , 


107! 1], 
| is K(A) © 9, so the 


large error in the rounding error corrupted solution is not due to ill-conditioning. Hence, Gaussian 
elimination without pivoting is numerically unstable. 

6. An Unstable Algorithm for Eigenvalue Computations: Finding the eigenvalues of a matrix by finding 
the roots of its characteristic polynomial is a numerically unstable process because the roots of the 
characteristic polynomial may be ill-conditioned when the eigenvalues of the corresponding matrix 
are well-conditioned. Transforming a matrix to companion form often requires an ill-conditioned 
similarity transformation, so even calculating the coefficients of the characteristic polynomial may 
be an unstable process. A well-known example is the diagonal matrix A = diag(1, 2,3,... , 20). The 
Wielandt-Hoffman theorem [GV96] shows that perturbing A to a nearby matrix A+ E perturbs 
the eigenvalues by no more than || E || -. However, the characteristic polynomial is the infamous 
Wilkinson polynomial discussed in Example 5 of Section 37.4, which has highly ill-conditioned 
roots. 


37-22 Handbook of Linear Algebra 


Author Note 


The contribution of Ralph Byers’ material is partially supported by the National Sciences Foundation 
Award 0098150. 


References 


[Bha96] R. Bhatia, Matrix Analysis, Springer, New York, 1996. 

[Bun87] J. Bunch, The weak and strong stability of algorithms in numerical linear algebra, Linear Algebra 
and Its Applications, 88, 49-66, 1987. 

[Dat95] B.N. Datta, Numerical Linear Algebra and Applications, Brooks/Cole Publishing Company, Pacific 
Grove, CA, 1995. (Section edition to be published in 2006.) 

[GV96] G.H. Golub and C.F. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press, 
Baltimore, MD, 1996. 

[Hig96] N.J. Higham, Accuracy and Stability of Numerical Algorithms, SIAM, Philadelphia, 1996. 

[HJ85] R.A. Horn and C.R. Johnson, Matrix Analysis, Cambridge University Press, New York, 1985. 

[IEEE754] IEEE 754-1985. “IEEE Standard for Binary Floating-Point Arithmetic” (ANSI/IEEE Std 754- 
1985), The Institute of Electrical and Electronics Engineers, Inc., New York, 1985. Reprinted in 
SIGPLAN Notices, 22(2): 9-25, 1987. 

[IEEE754r] (Draft) “Standard for Floating Point Arithmetic P754/D0.15.3—2005,” October 20. The In- 
stitute of Electrical and Electronics Engineers, Inc., New York, 2005. 

[IEEE854] IEEE 854-1987. “IEEE Standard for Radix-Independent Floating-Point Arithmetic.” The Insti- 
tute of Electrical and Electronics Engineers, Inc., New York, 1987. 

[Kat66] T. Kato, Perturbation Theory for Linear Operators, Springer-Verlag, New York, 1966. (A corrected 
second edition appears in 1980, which was republished in the Classics in Mathematics series in 1995.) 

[Ove01] M. Overton, Numerical Computing with IEEE Floating Point Arithmetic, Society for Industrial and 
Applied Mathematics, Philadelphia, PA, 2001. 

[Ric66] J. Rice, A theory of condition, SIAM J. Num. Anal., 3, 287-310, 1966. 

[Ste98] G.W. Stewart, Matrix Algorithms, Vol. 1, Basic Decompositions, SIAM, Philadelphia, 1998. 

[SS90] G.W. Stewart and J.G. Sun, Matrix Perturbation Theory, Academic Press, New York, 1990. 

[TB97] L.N. Trefethan and D. Bau, Numerical Linear Algebra, SIAM, Philadelphia, 1997. 

[Wil65] J.H. Wilkinson, The Algebraic Eigenvalue Problem, Clarendon Press, Oxford, U.K., 1965. 

[Wil64] J.H. Wilkinson, Rounding Errors in Algebraic Processes, Prentice-Hall, Inc., Upper Saddle River, NJ, 
1963. Reprinted by Dover Publications, Mineola, NY, 1994. 


38 


Matrix Factorizations 
and Direct Solution 
of Linear Systems 


38.1 Perturbations of Linear Systems .................... 38-2 

38.2 Triangular Linear Systems.................00...000. 38-5 

38.3. Gauss Elimination and LU Decomposition .......... 38-7 
Christopher Beatee 38.4 Orthogonalization and QR Decomposition ......... 38-13 
Viksinid Holpeclnio Tadein Ge wad Seats 38.5 Symmetric Factorizations ................. cece eee 38-15 
University References 33% 554.00 6c.5e. ee orcas, anos Aas omaeaRatee 38-17 


The need to solve systems of linear equations arises often within diverse disciplines of science, engineering, 
and finance. The expression “direct solution of linear systems” refers generally to computational strategies 
that can produce solutions to linear systems after a predetermined number of arithmetic operations that 
depends only on the structure and dimension of the coefficient matrix. The evolution of computers has 
and continues to influence the development of these strategies as well as fostering particular styles of 
perturbation analysis suited to illuminating their behavior. Some general themes have become dominant, 
as a result; others have been pushed aside. For example, Cramer’s Rule may be properly thought of as a 
direct solution strategy for solving linear systems; however it requires a much larger number of arithmetic 
operations than Gauss elimination and is generally much more susceptable to the deleterious effects of 
rounding. Most current approaches for the direct solution of a linear system, Ax = b, are patterned 
after Gauss elimination and favor systematically decoupling the system of equations. Zeros are introduced 
systematically into the coefficient matrix, transforming it into triangular form; the resulting triangular 
system is easily solved. The entire process can be viewed in this way: 


1. Find invertible matrices {5;}?_, such that S, ...S)S; A = U is triangular; then 
2. Calculate a modified right-hand side y = S, ... S. Sb; and then 
3. Determine the solution set to the triangular system Ux = y. 


The matrices S,, S2,... Sp, are typically either row permutations of lower triangular matrices (Gauss 
transformations) or unitary matrices and so have readily available inverses. Evidently A can be written 
as A = NU, where N = (S,...S2S,)~', A solution framework may built around the availability of 
decompositions such as this: 


1. Find a decompostion A = NU such that U is triangular and Ny = b is easily solved; 
2. Solve Ny = b; then 
3. Determine the solution set to triangular system Ux = y. 
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38.1 Perturbations of Linear Systems 


In the computational environment afforded by current computers, the finite representation of real numbers 
creates a small but persistant source of errors that may on occasion severely degrade the overall accuracy 
ofa calculation. This effect is of fundamental concern in assessing strategies for solving linear systems. 

Rounding errors can be introduced into the solution process for linear systems often before any calcula- 
tions are performed — as soon as data are stored within the computer and represented within the internal 
floating point number system of the computer. Further errors that may be introduced in the course of 
computation often may be viewed in aggregate as an effective additional contribution to the initial repre- 
sentation error. Inevitably then the linear system for which a solution is computed deviates slightly from 
the “true” linear system and it becomes of critical interest to determine whether such deviations will have 
a significant effect on the accuracy of the final computed result. 


Definitions: 


Let A € C”*” be a nonsingular matrix, b ¢ C”, and then denote by X = A~'b the unique solution of the 
linear system Ax = b. 

Given data perturbations 6A ¢ C"*” and 5b € C” to Aandb, respectively, the solution perturbation, 
5x € C” satisfies the associated perturbed linear system (A + 5.A)(X + 5x) = b + db (presuming then 
that the perturbed system is consistent). 

For anyx € C”, the residual vector associated with the linear system Ax = bis defined as r(x) = b— Ax. 

For any x € C”, the associated (norm-wise) relative backward error of the linear system Ax = b (with 
respect to the the p-norm) is 


there exist 5.A, db such that 


~ : 7 : |SAllp < Ell Allp 
np(A, b; x) = min < ¢ (A+ 6A)k = b+ bb with 


[Sb]lp < ellbllp 


forl<p<o. 
For any x € C”, the associated component-wise relative backward error of the linear system Ax = b 
is 
there exist 6 A, 5b such that 
|SA| < el Al 


A, b;X) = min < ¢ A+6A)x = b+ 6b with 
ee : - we [sb] < elb| 


where the absolute values and inequalities applied to vectors and matrices are interpretted component-wise: 
|B| < |A| means |b,| < |a;j| for all index pairs i, j. 
The (norm-wise) condition number of the linear system Ax = b (relative to the the p-norm) is 
IIbll p 
XII p 


Kp(A,&) = |A7"Ilp 


forl<p<o@. 
The matrix condition number relative to the the p-norm of A is 


Kp(A) = AllpllA“'llp 


forl<p<o. 
The Skeel condition number of the linear system Ax = b is 


Av Al Ik 
cond(A,2) = LIA UATE ileo 
IIXlloo 


The Skeel matrix condition number is cond(A) = || |A~!| | A] |loo- 
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Facts: [Hig96],[StS90] 


Ts 


a 


For any x € C”, X is the exact solution to any one of the family of perturbed linear systems 
(A + 6Ap)X = b+ dbp, 


where 6 € C, dbp = (8 — 1) r(x), 5Ag = Or(&)¥*, and y € C” is any vector such that 7*x = 1. In 
particular, for 9 = 0,6A = 0 and 6b = —r(X); for 6 = 1,5A = r(X)¥* and db = 0. 


. (Rigal—Gaches Theorem) For any x € C’”, 


7 Ir@llp 
ip As BSS) = 4 sy ae 
All pllXllp + IIbIlp 


Ify is the dual vector to x with respect to the p-norm (7*k = |l¥llq ||Xllp = 1 with +41 —1), then 
& is an exact solution to the perturbed linear system (A +6 Ag )X = b+ db, with data perturbations 


: 5 Ally Il 
as in (1) and@ = TAlp 1x1, +161, , and as a result 
b Ag dbz 
I9Asllp _ V8Bille 4 ee, 
|All, IIbllp 
(Oettli-Prager Theorem) For any x € C", 
Ir; | 


(A, b; x) = max - ; 
i ({A| |X| + [b]); 


vj 


If D; = diag Ia 
(JA] |x| + |b]); 
perturbed linear system (A + 6A)X = b + 5b with 6A = D, | A| D2 and 5b = —D, |b| 


) and D,) = diag(sign(x);), then % is an exact solution to the 


|SA| < @(A,b;x)|A] and [db] < @(A, b; x) | A] 


and no smaller constant can be used in place of w(A, b; x). 
The reciprocal of xp (A) is the smallest norm-wise relative distance of A to a singular matrix, ie., 


1 ae 
= min 
Kp(A) Alp 


In particular, the perturbed coefficient matrix A + 6A is nonsingular if 


A+dAis singular, 


IlSAll, 1 
< : 
|Allp — &p(A) 


1 < Kp(A,&) < Kp(A) and 1 < cond(A,%) < cond(A) < k(A). 
cond(A) = min { Koo(D A) | D diagonal ie 


If5A = 0, then 
6 bb 
l xl» < «p( A,X) I Ip 
Xl p I|bII , 
If 5b = O and A + 6A is nonsingular, then 
5 ébA 
| x|lp Zi! Ip 
|X + 4x p |All 


1 
Tf SAllp < €l|Allp, lobllp < €llbllp, ande < ——,, then 
P P P P kp(A) 


[dxllp — _2€Kp(A) 
[Xp — 1—€xp(A) 
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1 
10. If |S A| < €|Al, |db| < € |b], and ¢€ < ——__., then 
Cc 


ond(A) 


]5XIloo L 2€ cond(A, %) 
Ilo ~ 1—«€cond(A) 


Examples: 


1. Let A = 


999 998 999  —1000 
K,(A) © 3.996 x 10°. Consider 


1999 1 
b= associated with a solution X = | 


1000 999 ‘ —998 999 
Al= . Then ||All; = |A7'I], = 1999 so that 


1997 


A perturbation of right-hand side 5b = ena constitutes a relative change in the right-hand 


side of 1 ~ 5.005 x 10-6 yet it produces a perturbed solution X + 6x = | 


- constitutin 
Illi g 


—18.99 
a relative change an = 19.98 < 20 = (A) oe The bound determined by the condition 
number is very nearly achieved. Note that the same perturbed solution x + 6x could be produced 


by a change in the coefficient matrix 


5A =iy* ae ! : (1/3996) aa 
= = 0.01| 139.96 39.96 = 4 


constituting a relative change ae ~~ 2.5 x 10-7. Then (A + 5 A)(X + 6x) = b. 


. Let n = 100 and A be tridiagonal with diagonal entries equal to —2 and all superdiagonal and 


subdiagonal entries equal to 1 (associated with a centered difference approximation to the second 
derivative). Let b be a vector with a quadratic variation in entries 


by = (k — 1)(100 — k)/10, 000. 
Then 
k)(A,X) 1, but (A) © 4.1336 x 10°. 


Since the elements ofb do not have an exact binary representation, the linear system that is presented 
to any computational algorithm will be Ax = b + 5b with ||Sb||2 < €||b||2, where € is the unit 
roundoff error. For example, if the linear system data is stored in IEEE single precision format, 
€ © 6 x 10-®. The matrix condition number, «(A), would yield a bound of (6 x 107°)(4.1336 x 
10°) © 2.5 x 10~4 anticipating the loss of more than 4 significant digits in solution components 
even if all computations were done on the stored data with no further error. However, the condition 
number of the linear system, (A, %), is substantially smaller and the predicted error for the system 
is roughly the same as the initial representation error © 6 x 107°, indicating that the solution will 
be fairly insensitive to the consequences of rounding of the right-hand side data—assuming no 
further errors occur. But, in fact, this conclusion remains true even if further errors occur, ifwhatever 
computational algorithm that is used produces small backward error, as might be asserted if, say, 
a final residual satisfies ||r||2 < O(€) ||b||2. This situation changes substantially if the right-hand 
side is changed to 


b, = (—1)*(k — 1)(100 — k)/10, 000, 


Matrix Factorizations and Direct Solution of Linear Systems 38-5 


which only introduces a sign variation in b. In this case, «2(A,X) © K2(A), and the components of 
the computed solution can be expected to lose about 4 significant digits purely on the basis of errors 
that are made in the initial representation. Additional errors made in the course of the computation 
can hardly be expected to improve this situation. 


38.2 Triangular Linear Systems 


Systems of linear equations for which the unknowns may be solved for one at a time in sequence may 
be reordered to produce linear systems with triangular coefficient matrices. Such systems can be solved 
both with remarkable accuracy and remarkable efficiency. Triangular systems are the archetype for easily 
solvable systems of linear equations; as such, they often constitute an intermediate goal for strategies of 
solving linear systems. 


Definitions: 


A linear system of equations Tx = b with T € C”*" (representing m equations in n unknowns) is a 
triangular system if T = [t,] is either an upper triangular matrix (tj; = 0 fori > j) or a lower 
triangular matrix (t;, = 0 fori < j). 


Facts: [Hig96], [GV96] 
1. [GV96, pp. 88-90] 


Algorithm 1: Row-oriented forward-substitution for solving lower triangular system 
Input: L € R’*” with €; = 0 fori < j;b € R” 
Output: solution vector x € R" that satisfies Lx = b 


x) <— by /£11 
Fork =2:n 


XE (bE — Liik—1 * X1:k-1)/lek 


end 


2. [GV96, pp. 88-90] 


Algorithm 2: Column-oriented back-substitution for solving upper triangular system 
Input: U € R"*” with uj; = 0 fori > j;b € R" 
Output: solution vector x € R” that satisfies Ux = b 
For k = n down to 2 by steps of —1 
Xk <— De /UK,k 
Bysk—-1 << Bik—1 — XK 1-1 
end 


x) <— by/uy,1 


3. Algorithm 1 involves as a core calculation, dot products of portions of coefficient matrix rows with 
corresponding portions of the emerging solution vector. This can incur a performance penalty for 
large n from accumulation of dot products using a scalar recurrence. A “column-oriented” refor- 
mulation may have better performance for large n. Algorithm 2 is a “column-oriented” formulation 
for solving upper triangular systems. 
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4. The solution of triangular systems using either Algorithm 1 or 2 is componentwise backward stable. In 
particular the computed result, x, produced either by Algorithm 1 or 2 in solving a triangular system, 


Tx = b, will be the exact result of a perturbed system (T + 5T)x = b, where |ST| < ; |T| 
—ne 


and € is the unit roundoff error. 
5. The error in the solution of a triangular system, Tx = b, using either Algorithm 1 or 2 satisfies 
IX — Xlloo s ne cond(T, xX) 
Xl ~ 1—ne(cond(T) +1) 


6. If T = [#] is an lower triangular matrix satisfying |t;;| > |t;;| for j < i, the computed solution to 
the linear system Tx = b produced by either Algorithm 1 or the variant of Algorithm 2 for lower 
triangular systems satisfies 


i 


Iki — %i1 < max |X;|, 
l—ne js<i 


where X; are the components of the computed solution, x, and %; are the components of the 
exact solution, x. Although this bound degrades exponentially with 7, it shows that early solution 
components will be computed to high accuracy relative to those components already computed. 


Examples: 


1. Use Algorithm 2 to solve the triangular system 


jae eee i 1 
—6 x2) = 1 
3 x3 1 


k = 3 step: Solve for x3 = 1/3. Update right-hand side: 


1 2 1 3 2 
x] 
0 al [R=]: (1/3) |-6| = |3]. 
x2 
0 O 1 0 
k = 2 step: Solve for x. = 3/2. Update right-hand side: 
1 2 -—l 
Of} fea] SS) B22) | 0). 
0 0 0 0 
k = 1 step: Solve for x; = —1. 
1 0 0 1 0 O 
2. ({Hig96, p. 156]) Fore > 0, consider T= |e ¢ O|.Then TT! = |-1 : 0}, and so 
011 1-1 1 
cond(T) = 5, even though 
1 2 
Koo(T) = 2(2 + pe ae + O(1) 


Thus, linear systems having T as a coefficient matrix will be solved to high relative accuracy, 
independent of both right-hand side and size of €, despite the poor conditioning of T (as measured 
by Koo) as € becomes small. However, note that 


2 2 2 
cond(T’)=1+— and x«,(T')=(1te)- + —4+O(1). 
€ E E 
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So, linear systems having T? as a coefficient matrix may have solutions that are sensitive to per- 
turbations and indeed, cond(T’, x) * cond(T*) for any right-hand side b with b; # 0 yielding 
solutions that are sensitive to perturbations for small €. 


38.3 Gauss Elimination and LU Decomposition 


Gauss elimination is an elementary approach to solving systems of linear equations, yet it still constitutes 
the core of the most sophisticated of solution strategies. In the kth step, a transformation matrix, M;, 
(a “Gauss transformation”) is designed so as to introduce zeros into A — typically into a portion of the 
k‘" column — without harming zeros that have been introduced in earlier steps. Typically, successive 
applications of Gauss transformations are interleaved with row interchanges. Remarkably, this reduction 
process can be viewed as producing a decomposition of the coefficient matrix A = NU, where U isa 
triangular matrix and N is a row permutation of a lower triangular matrix. 


Definitions: 
For each index k, a Gauss vector is a vector in C” with the leading k entries equal to zero: £, = 
[0,...,0,€z41,...,£n]7. The entries €,4,...,£, are Gauss multipliers and the associated matrix 
SY’ 
k 
M, = I — kez 


is called a Gauss transformation. 

For the pair of indices (i, j), with i < j the associated permutation matrix, [];,; isan n x n identity 
matrix with the i” row and j‘" row interchanged. Note that 7;, is the identity matrix. 

A matrix U € C”*" is in row-echelon form if (1) the first nonzero entry of each row has a strictly 
smaller column index than all nonzero entries having a strictly larger row index and (2) zero rows 
occur at the bottom. The first nonzero entry in each row of U is called a pivot. Thus, the deter- 
mining feature of row echelon form is that pivots occur to the left of all nonzero entries in lower 
rows. 

A matrix A € C”*” has an LU decomposition if there exists a unit lower triangular matrix L € C”*” 
(Li; = 0 fori < j and L;; = 1 for alli) and an upper triangular matrix U € Cr (U;,; = Ofori > j) 
such that A = LU. 


Facts: [GV96] 


1. Leta € C” bea vector with a nonzero component in ther th entry, a, 4 0. Define the Gauss vector, 
£,-= [052...50; er oaks ae The associated Gauss transformation M, = I — Let introduces 
i jar + 


r 
zeros into the last n — r entries of a: 
_ T 
M,a= [a1,..., a,, 0,..., OJ. 


2. If Ae C”™*” with rank(A) = p > 1 has p leading principal submatrices nonsingular, Aj..1.., 7 = 
1, ..., p, then there exist Gauss transformations M;, Mo, ..., M, so that 


Mp My-1 ees M,A — U 


with U upper triangular. Each Gauss transformation M, introduces zeros into the r‘" column. 
3. Gauss transformations are unit lower triangular matrices. They are invertible, and for the Gauss 
transformation, M, = I — £,e7, 


My! =1+4£,e7. 


38-8 


4. 


5. 


10. 
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If Gauss vectors £), £2, ..., &,—1 are given with 
0 0 0 
ly) 0 0 


ent ln Lani 


then the product of Gauss transformations M,—;My—2--- MM, is invertible and has an explicit 
inverse 


1 0 0 0 

ae | 0 
n—1 

(My—1My-2...MoMi)"! =I+ Sef = [os bn 0 
k=1 

1 0 

Lunt Lng tee Lan—1 1 


If A € C”*" with rank(A) = ¢ has p leading principal submatrices nonsingular, Aj..1..,7 = 
1, ..., p, then A has an LU decomposition: A = LU, with L unit lower triangular and U upper 
triangular. The (i, j) entry of L, L;,; withi > j is the Gauss multiplier used to introduce a zero into 
the corresponding (i, 7) entry of A. If, additionally, o = m, then the LU decomposition is unique. 
Let a be an arbitrary vector in C”. For any index r, there is an index x > 7, a permutation matrix 
Il,,,, and a Gauss transformation M, so that 


M,11,,,a= [4], ..-5 Gr—1, Ap, 0, -.-5 o}". 
4 
n—-r 

The index jz is chosen so that a,, # 0 out of the set {a,, 41, ..., dn}. Ifa, #0, then wp = k and 
II,,,, = I is a possible choice; if each element is zero, a, = a, +1 oon ad, = 0, then uw = k, 
II,,, = I, and M, = I isa possible choice. 

For every matrix A € C”*” with rank(A) = p, there exists a sequence of p indices 11, [2, ..., Lp 
with? < uw; < m fori =1,..., and Gauss transformations M), Mo, ..., M, so that 


Mp M1 p,u,Mp—-1 1 p-1,0,-1 i ~My, A =U 


with U upper triangular and in row echelon form. Each pair of transformations M, IT,,,,, introduces 
zeros below the r’ pivot. 

Forr <i <j, 1,;M, = MMi j, 
of £, are interchanged to form @, ). 
For every matrix A € C”*” with rank(A) = p, there is a row permutation of A that has an 
LU decomposition: PA = LU, with a permutation matrix P, unit lower triangular matrix 
L, and an upper triangular matrix U that is in row echelon form. P can be chosen as P = 
Tu, p—1,u,-1 +++ Miu, from (7), though in general there can be many other possibilities as well. 
Reduction of A with Gauss transformations (or equivalently, calculation of an LU factorization) 
must generally incorporate row interchanges. As a practical matter, these row interchanges com- 
monly are chosen so as to bring the largest magnitude entry within the column being reduced up 
into the pivot location. This strategy is called “partial pivoting.” In particular, if zeros are to be 
introduced into the k™" column below the r™ row (with r < k), then one seeks an index jz, such 
that r < pw, < mand |A,, «| = max,<j<m|Aix|. When (41, flo, ..., Wp, in (7) are chosen in this 
way, the reduction process is called “Gaussian Elimination with Partial Pivoting” (GEPP) or, within 
the context of factorization, the permuted LU factorization (PLU). 


where M, =![- ber and @, = 11;,;4 (ie., thei and j entries 
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11. [GV96, p. 115] 


Algorithm 1: GEPP/PLU decomposition of a rectangular matrix (outer product) 


Input: A € R”*” 
Output: L €¢ R”*” (unit lower triangular matrix) 
U € R”*" (upper triangular matrix - row echelon form) 
P €R”™*” (permutation matrix) so that PA = LU 
(P is represented with an index vector p such that y = Pz yj = Zp,) 
Le In;U —0€R™' 
p = [1, 2, 3,..., m] 
r<l; 
Fork = 1ton 
Find yz such thatr < uw < mand |A,,4| = max, <j<m | Ail 
If Aye # 0, then 
Exchange Aykn <> Arkens Lug <> Lrir—1, and py <> py 
Ly sismyr <= Ar +iimk/ Ark 
Un ken = Arlen 
Fori=r+ltom 
For j =k+1ton 
Ai,j <— Ai,j _ Li U;,j 
r<r+l 


12. [GV96, p. 115] 


Algorithm 2: GEPP/PLU decomposition of a rectangular matrix (gaxpy) 
Input: A € R”*" 
Output: L € R”*” (unit lower triangular matrix), 
U € R”*” (upper triangular matrix - row echelon form), and 
P €R”*” (permutation matrix) so that PA = LU 
(P is represented with an index vector 7 that records row interchanges 
1, = (££ Means rowr and row jt > r were interchanged) 
L<I, €R"*";U —0€R”™”;andr < 1; 
For j =1ton 
v<— Aim, 
Ifr > 1, then 
fori =1ltor —1, 
Exchange vj = Vv z, 
Solve the triangular system, Ly.-—11..-1 + Z = Vir—13 
Uj -1,; <— Z; 
Update Vy:m << Veen — Lrimar—1° ZB 
Find yz such that |v,,| = max, <j<m |vi| 
If v,, 4 0, then 
Ty <p 
Exchange v,, <> v; 
Fori = 1ltor—1, 
Exchange Lj,j <> Lj 
Lye +timr = Vetim/Vr 
U,,j <— Vr 
r<rt+l 


13. The condition for skipping reduction steps (A, 4 0 in Algorithm 1 and v,, # 0 in Algorithm 2) 
indicates deficiency of column rank and the potential for an infinite number of solutions. These 
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conditions are sensitive to rounding errors that may occur in the calculation of those columns 
and as such, GEPP/PLU is applied for the most part in full column rank settings (rank(A) = n), 
guaranteeing that no zero pivots are encountered and that no reduction steps are skipped. 
14. Both Algorithms 1 and 2 require approximately =p? +pm(n—p)+pn(m—p) arithmetic operations. 
Algorithm 1 involves as a core calculation the updating of a submatrix having ever diminishing size. 
For large matrix dimension, the contents of this submatrix, A;+1:m,t+1:n. may be widely scattered 
through computer memory and a performance penalty can occur in gathering the data for compu- 
tation (which can be costly relative to the number of arithmetic operations that must be performed). 
Algorithm 2 is a reorganization that avoids excess data motion by delaying updates to columns until 
the step within which they have zeros introduced. This forces modifications to the matrix entries 
to be made just one column at a time and the necessary data motion can be more efficient. 
15. Other strategies for avoiding the adverse effects of small pivots exist. Some are more aggressive than 
partial pivoting in producing the largest possible pivot, others are more restrained. 
“Complete pivoting” uses both row and column permutations to bring in the largest possible 
pivot: If zeros are to be introduced into the kt column in row entries r + 1 to m, then one seeks 
indices and v such thatr < uw < mandk < v < nsuch that |A,y| = MaX ism | Aj;,;|. Gauss 


elimination with complete pivoting produces a unit lower triangular matrix L € R”*"”, an upper 
triangular matrix U € R”*", and two permutation matrices, P and Q, so that PAQ = LU. 
“Threshold pivoting” identifies pivot candidates in each step that achieve a significant (prede- 

termined) fraction of the magnitude of the pivot that would have been used in that step for partial 
pivoting: Consider all 2 such thatr < f < mand |Agx| > T - max,<j<m|Aix|, where tT € (0,1) 
is a given threshold. This allows pivots to be chosen on the basis of other criteria such as influence 
on sparsity while still providing some protection from instability. t can often be chosen quite small 
(t = 0.1 or tT = 0.025 are typical values). 

16. If P e R"™*”, £ e R”™*", and U € R™*” are the computed permutation matrix and LU factors 
from either Algorithm 1 or 2 on A € R”*", then 


ne Beas oo I 
|L||U| 


on 4 2 
LU=P(A+6A) with |éA| < 
l—ne 


and for the particular case that m = n and A is nonsingular, if an approximate solution, %, to 
Ax = b is computed by solving the two triangular linear systems, Ly = Pb and Ux = y, then x is 
the exact solution to a perturbed linear system: 


ne »« 


(A+ 5A)k=b with [SA < PT ILO). 


l—ne 
Furthermore, |Lj,;| < 1 and |Uj,;| < 2-1 maxpg; | Ax,j|, 80 


NyreE 


2 
ISAlloo S 5 I| Alloo- 


— ne 


Examples: 


1. Using Algorithm 1, find a permuted LU factorization of 


N 

N 

iN 
Re FB OO WN 
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Setup: p=([1234],r<1 
k = 1 step: be<—2p= (2134) 
2 2 6 
1 3 
Permuted A: 
x eee | 
1 1 1 
1 0 0 0 22 4 6 
3; 100 00 0 0 
LU snapshot: L= Fi and U= ‘ 
57). 02 ve <0 0: DG 
3 001 000 0 
00 =f 
Updated A2:4,2:4: 01 
OG aly 2 
r<2 
k = 2 step: <= 2,|A22| = maxz<j<a |Ai2| = 0 
k = 3 step: bm <— 3,p = [2314], |A33] = maxr<j<4|Ai3| = 1 
1 4 
Permuted Aj.43.4: ]0 —1 
oe 
100 0 2 2 4 6 
LU snapsh re |e eee ee eel (ee 
snapshot: = an = a 
e 3 010 000 0 
5 101 000 0 
1 
Updated A3:4,42 | 
reo 
k =4 step: bw <— 4p = [2341], |Agal = max3<j<a |Aial = 6 
—6 
Permuted Ao.43:4: : 
LO © 0 22 4 6 
3 1 0 0 001 4 
LU snapshot: L= 7 and U= 
5 1 10 0 0 0 6 
1 1 
5 9 7 1 0 0 0 0 
The permutation matrix associated with p = [2 3 4 1] is 
0 1 0 0 
0 0 1 0 
P= 
0 0 0 1 
100 0 
and 
2 2 4 6 B20: 02"-0)))- (F2> 2° A 
ee Sk ay A -; 10 0 Dal SAN! Se Se 
et BP ae IT Be a SOR Os Orage 
1 1 
I dh 22 > oz 1] [0 00 oO 
2. Using Algorithm 2, solve the system of linear equations 
ti 23 at) Peg 1 
Fe oe cf) Ng er |S 
yes ee Be 3 
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Phase 1: Find permuted LU decomposition. 


r<—1 
1 2 
j = step: ve |2). 7, — pw =2. Permutedv: | 1 
2 2 
1 0 0 2. QD 
LU snapshot: m=([2]) L= 3 1 0 and U=]|0 0 0 
LG A 0 0 0 
r<2 
3 2 
j =2 step: ve 2| . Permuted v: 3 
-1 aI 
Solve 1- z= 2.U,2 <— z= [2]. 
v 2 3 7 
ee = =i|2 | 
v3 3 =] 1 
my <— pw = 3.132 <— —4 and U2 <— —3 
1 0 0 a 2.74 
LU snapshot: m= [2,3] L=j 1 1 0 and U=]|0 -3 0 
1 2 
ee | 0 00 
r<—3 
=i 
: 10 eI 
j = 3 step: v< |—1] Permuted v: 0} . Solve ‘Z= > 
0 i Lei 0 
U; -l =I 
) as .v3 24 =1-[5, -4]- 
U23 1 1 
m3 — 3 and U33 <2t 
1 0 0 2 2v =| 
LU snapshot: mw = (2, 3,3] L=j 1 1 0 and U=j0 -3 1 
1 2 1 
1-2 1 0 0 24 
0 1 0 
The permutation matrix associated with 7 is P= |0 0 1] and 
1 0 0 


0 
PA=]2 -1 oO; =]1 1 0} Jo -3 1) =L-U. 
boo dl a foo al 


1 0 0 yal 3 
1 1 0 yo| = 3 > Vv 3, y2 6, V3 6= 
ee ame 1 


Phase 3: Solve the upper triangular system Ux = y. 


2 2 - 


1 x1 —3 

0 -3 1 x»! = 6 > x, =1, x2 1, x3 = 3. 
1 1 
6 2 


0 oO 22] |x; 6 
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38.4 Orthogonalization and QR Decomposition 


The process of transforming an arbitrary linear system into a triangular system may also be approached 
by systematically introducing zeros into the coefficient matrix with unitary transformations: Given a 
system Ax = b, find unitary matrices Vi, V2, --- Vz such that Vy... V)V, A = T is triangular; calculate 
y = Ve--- Vo V\b; solve the triangular system Tx = y. 

There are two different types of rudimentary unitary transformations that are described here: House- 
holder transformations and Givens transformations. 


Definitions: 


Let v € C” bea nonzero vector. The matrix H = I — mee is called a Householder transformation 
(or Householder reflector). In this context, v, is called a Householder vector. 

For 0, 0 € [0, 27), let G(i, j,0,%) be an n x n identity matrix modified so that the (i,7) and (j, j) 
entries are replaced by c = cos(@), the (i, j) entry is replaced by s = e’® sin(@), and the (j,i) entry is 


replaced by —§ = —e~"” sin(@): 


1 0 0 0 

0 Cc s 0 
G(i, 49; v) =a 

0 —§ Cc 0 

0 0 0 1 


G(i, j,0, 0) is called a Givens transformation (or Givens rotation). 


Facts: [GV96] 


1. Householder transformations are unitary matrices. 


2. Leta € C” bea nonzero vector. Define v = sign(a,)|lalle; +-a with e; = [1,0,..., 0]? € C". Then 
the Householder transformation H = I — he satisfies 
Vilo 
Ha=ae, with a = —sign(a;)|lall. 


3. [GV96,pp. 210-213] 


Algorithm 3: Householder QR Factorization: 


Input: matrix A € C”*" with m > n 
Output: the QR factorization A = QR, where the upper triangular part of R is stored in the 
upper triangular part of A 
Q=In 
Fork =1:n 
x= Ak:mk 
vj = sign(x;)||x\le: +x, where e, ¢ C” **! 
Vi = Vk/IIVell 
Ak:men — (lin—k-+41 =| 2ViV,) Akimkin 
Qi:k-1k:m = Qi :k—14k:m Tink 1 = 2vEV;) 
Qk:mkim — Qk:mkim Tin—k-+1 _ 2VEV;) 
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4, [GV96, p. 215] A Givens rotation is a unitary matrix. 
5. [GV96, pp. 216-221] For any scalars x, y € C, there exists a Givens rotation G € C?*? such that 


-L JE-E) 


where c, s, andr can be computed via 
° If y = 0 (includes the case x = y = 0), thenc =1,5=0,r=x. 
* Ifx =0(y must be nonzero), then c = 0, s = sign(j), r = ly. 
* If both x and y are nonzero, then c = |x|/./|x/? + ly/?, 

s= sign(x)7/./|xP + ly? r = sign(x)/ |x|? + ly|?. 


6. [GV96, pp. 226-227] 


Algorithm 4: Givens QR Factorization 
Input: matrix A € C”*" with m > n 
Output: the QR factorization A = QR, where the upper triangular part of R is stored in the 
upper triangular part of A 
Q=Imn 
Fork =1:n 

Fori=k+1:m 

[x,y] = [Axes Aik] 


=S°. ¢ 


Ak kin A skin 
ae kk 
Aiken Aiken 


[ Quemks Qi-mi] —_ [ Qi:mk> Qi:mi ]G* 


Ets 
ComputeG =| _ | via Fact 5. 


7. [GV96, p. 212] In many applications, it is not necessary to compute Q explicitly in Algorithm 3 
and Algorithm 4. See also [TB97, p. 74] for details. 

8. [GV96, pp. 225-227] If A € R”*" with m > n, then the cost of Algorithm 3 without explicitly 
computing Q is 2n?(m — n/3) flops and the cost of Algorithm 4 without explicitly computing Q 
is 3n?(m — n/3) flops. 

9. [Mey00, p. 349] Algorithm 3 and Algorithm 4 are numerically stable for computing the QR factor- 
ization. 


Examples: 


1 1 
1. Weshall use Givens rotations to transform A = F 2) toupper triangular form, asin Algorithm 4. 
1 3 
1) 


First, to annihilate the element in position (2,1), we use Fact 5 with (x,y) = (1,1) and obtain 


c=s= 1/V2; hence: 


0.7071 0.7071 oO] f1 1 1.4142 2.1213 
A” = G,A = |—0.7071 0.7071 0] |1 2] = Jo 0.7071 
0 0 1] J1 3 1 3 
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Next, to annihilate the element in position (3,1), we use (x, y) = (1.4142, 1) in Fact 5 and get 


0.8165 0 ed jee 3.4641 
A? = G)A = 10 1 0 AY = 10 0.7071] . 
eae 0 ime i ae 


Finally, we annihilate the element in position (3,2) using (x, y) = (.7071, 1.2247): 


tcp 0 1.7321 3.4641 
A? =G;A? = |0 0.5000 0.8660] A? = }0 1.4142 
0 —0.8660 0.5000 0 0 


Asa result, R = A® and R consists of the first two rows of A®), The matrix Q can be computed 
as the product G7?G?G?. 


2. We shall use Householder reflections to transform A from Example 1 to upper triangular form 
as in Algorithm 3. First, let'a = A.) = [1 1 i, y= —-V3,a= [-v3 0 0], and 
u; = [0.8881 0.3251 0.3251] "; then 


Quy A 
_—_—_~—__ [= 1.7321 — 3.4641 
A® = (I-2uuy) A = A—u, [3.0764 5.0267] = | 0 0.3660 
0 1.3660 
@) . 

Next, 2 = —||A33||2, u2 = [0 0.7934 0.6088] ", and 

2uy A 
—~—~__ [=1.7321  —3.4641 
A? = (I —2muz) AY = AM — uw [0 2.2439] = | 0 —1.4142 


0 0 


Note that R = A®) has changed sign as compared with Example 1. The matrix Q can be computed 
as (I _ 2uut ) (I _ 2u,ut y Therefore, we have full information about the transformation if we 
store the vectors u, and u». 


38.5 Symmetric Factorizations 


Real symmetric matrices (A = A’) and their complex analogs, Hermitian matrices (Chapter 8), are 
specified by roughly half the number of parameters than general n x n matrices, so one could anticipate 
benefits that take advantage of this structure. 


Definitions: 


Ann x n matrix, A, is Hermitian if A = A* = A’. 

A € C”" is positive-definite if x* Ax > 0 for allx € C” withx 4 0. 

The Cholesky decomposition (or Cholesky factorization) of a positive-definite matrix A is A = G G* 
with G € C"*” lower triangular and having positive diagonal entries. 


Facts: [Hig96], [GV96] 


1. A positive-definite matrix is Hermitian. Note that the similar but weaker assertion for a matrix 
A € R"*" that “x! Ax > 0 for all x € R” with x 4 0” does not imply that A = A’. 
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2. If A € C"*" is positive-definite, then A has an LU decomposition, A = LU, and the diagonal of 
U, {uy1, W225 ..-5 Unn}, has strictly positive entries. 

3. If A € C"*" is positive-definite, then the LU decomposition of A satisfies A = LU with U = D L* 
and D = diag(U). Thus, A can be written as A = LDL* with L unit lower triangular and D 
diagonal with positive diagonal entries. Furthermore, A has a Cholesky decomposition A = G G* 
with G € C"*" lower triangular. Indeed, if 


D = diag ({./un, Jr; oe) VUnn}) 


then DD = DandG =LD. 
4. [GV96, p. 144] The Cholesky decomposition of a positive-definite matrix A can be computed 
directly: 


Algorithm 1: Cholesky decomposition of a positive-definite matrix 


Input: A € C"*” positive definite 
Output: G € C”*" (lower triangular matrix so that A = G G*) 
G<0eEC"™"; 
For j = 1lton 
v<— Ajin,j 
fork = 1toj—1, 


V <—V— GKG jen 


G j:n,j — Vn 


5. Algorithm 1 requires approximately n° /3 floating point arithmetic operations and n floating point 
square roots to complete (roughly half of what is required for an LU decomposition). 
6. If A € R"*" is symmetric and positive-definite and Algorithm 1 runs to completion producing a 
computed Cholesky factor G € R”™", then 
AA 1 eK 
GOTH A45A with jaaj<e “4 Y© J@1 67), 
l1—-(n+l)e 
Furthermore, if an approximate solution, %, to Ax = b is computed by solving the two triangular 
linear systems Gy = b and G?X = y, and a scaling matrix is defined as A = diag(,/aj;), then the 
scaled error A(x — X) satisfies 


IA — X)Il2 . ky(H)e 
WAxll2 ~ 1—K2(H)e’ 


where A = AHA. If k.(H) & (A), then it is quite likely that the entries of AX will have 
mostly the same magnitude and so the error bound suggests that all entries of the solution will be 
computed to high relative accuracy. 

7. If A € C”*” is Hermitian and has all leading principal submatrices nonsingular, then A has an 
LU decomposition that can be written as A= LU = L D L* with L unit lower triangular and D 
diagonal with real diagonal entries. Furthermore, the number of positive and negative entries of D 
is equal to the number of positive and negative eigenvalues of A, respectively (the Sylvester law of 
inertia). 

8. Note that it may not be prudent to compute the LU (or LDL’) decomposition of a Hermitian 
indefinite matrix A without pivoting, yet the use of pivoting will likely eliminate the advantages 
symmetry might offer. An alternative is a block L DL’ decomposition that incorporates a diagonal 
pivoting strategy (see [GV96] for details). 
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Examples: 
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1. Calculate the Cholesky decomposition of the 3 x 3 Hilbert matrix, 


WIE NI Re 


BIE Ble Dele 


Ur Ble le 


Setup: 


j = step: 


G snapshot: 


v<[l, 5 
1 
G=]3 
1 
3 


j =2 step: 


G snapshot: 


j = 3 step: 


G snapshot: 


1 1 
Ves ( 
1 0 0 
1 1 
Gata. og) So 
I Shyy i, 
3 273 65 


” 
Nu 
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39.1 Basic Concepts 


(See Chapter 5 for additional information.) 


Definitions: 


Given a vector b € R” and a matrix A € R”*" with m > n, the least squares problem is to find a vector 
Xo € R” that minimizes the Euclidean length of the difference between Ax and b: 


Problem LS: Find x satisfying ||b — Axo||.2 = min ||b — Ax|l2. 


Such an Xp is called a least squares solution. 

For any vector x the vector r = r(x) = b — Ax is the residual vector. The residual of a least squares 
solution is denoted by ro. The least squares problem is consistent if b € range(A). 

A basic solution, xog, is a least squares solution with at least n — rank(A) zero components. The 
minimum-norm least squares solution, xoy, is the least squares solution of minimum Euclidean norm. 

In a weighted least squares problem, we are also given weights w; > 0 fori = 1,...,m, and the 
objective is to minimize || W(b — Ax)||2, where W = diag(w),... , Wi). This is an important special case 
of the generalized least squares problem: 


Problem GLS: Let Ax + Bv = b, whereB € R””? with p < m. 
Find xc, vc such that ||v||2 is minimized. 


Note that Bv plays the role of the residual vector. 
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In the total least squares problem, we allow for errors both in the vector b and in the matrix A: 


Problem TLS: Let (A + E)x+r=b,where E € R”*” withn < m. 
Find xy such that ||(E,r)|| ¢is minimized. 


|| - lz denotes the Frobenius norm. 

In this chapter, A;; and A,; denote the vectors given by the elements in the ith row and jth column 
of matrix A, respectively. Similarly, A,.,, (or A, ) is the submatrix consisting of rows (or columns) k 
through / of A. 

In the examples in this chapter, the computation is done with about 16 digits accuracy, but the displayed 
results are rounded to fewer digits. 


Facts: 
(See, e.g., Chapters 1 and 2 in [Bjo96].) 


1. Ifm > n = rank(A), then the least squares solution xo is analytically equivalent to the solution to 
the normal equations A'Ax = Ab. 
2. If the least squares problem is consistent, then ro = 0. 
3. The least squares solution is unique if m > n and A has full rank. 
4. Ifthe system is underdetermined (m < n) or if A is rank deficient, then the solution to Problem LS 
is not unique. Also, a basic solution is not unique. 
5. The minimum-norm least squares solution xox is always unique. 
6. Ifm > n= rank(A), then the least squares solution can be written as Xx» = A‘b, where the matrix 
At is the Moore—Penrose generalized inverse or pseudoinverse of A. (See Section 5.7.) In general, 
At produces the minimum-norm solution: xy = A‘b. 
7. If B is nonsingular, then xg minimizes || B~!(b — Ax)|l2. 
8. If the covariance matrix for b has the Cholesky factorization Cov(b) = C TC, then xg is the best 
linear unbiased estimate (BLUE) in the general linear model with B = C a 
9. If Cov(b) has full rank, then xg is the solution to the least squares problem min ||C~7 (b — Ax)||>. 
10. In particular, if Cov(b) = o7I, then xg = Xo and Cov(xo) = o7(A™A)1. 
11. An English translation of the original work on least squares problems by C. F. Gauss is available 
in [Gau95]. 


Examples: 


1. Consider problem LS with m = 3 and n = 2: 
1 1 0.75 
min|||1 2 | | — |1.13 
1.39 
The associated normal equations and the least squares solution are 
3 6] |x| _ [3.27 «048 
6 14] |x} 17.18] Or 10321 ° 
The residual vector corresponding to Xp is 
—0.02 


fo = r(x) =b AXo = 0.04) , 
—0.02 


and A’ ry = 0. 
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2. If we use the weights w; = 10 and w2 = w3 = 1, the problem is changed to 
10 10 7.5 
min||| 1 2 a — }1.13 
To < 35) 1.39 
whose least squares solution and corresponding residual are 


be ; pater 
= oy o= . 
0.33186 ieee 


Note that the first component of rp is reduced when w is increased from 1 to 10. 


39.2 Least Squares Data Fitting 
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Definitions: 


Given m data points (f;, y;),i = 1,...,m, and n linearly independent functions f;, j = 1,..., 


m > n), find the linear combination 


Fan) =. x F,@) 


j=l 


that minimizes the sum of squared residuals y; — F (x, t;) at the data points: 


min Uy ; — F (x,t) 


n (with 


The coefficients x; are the components of the least squares solution to min ||b — Ax||2, where the columns 


A, of A are samples of f; at t; and the elements of b are the values y;: 
Aij = fi (&), bi = yi, i=1,...,m f SH l\yses gh. 
The solution F (Xo, t) is said to fit the data in the least squares sense. 


Facts: 
(See, e.g., Chapter 4 in [Bjo96].) 


1. The fit can be made more robust to outliers by solving a weighted least squares problem 
min || W(b — Ax)||2 with W = diag(wj,...,Wm), wi = W(ri) = W(b; — Aj;;x); w being a convex 


function. This problem is usually solved by an iteratively reweighted least squares algorithm. 


2. In orthogonal distance fitting, instead of minimizing the residuals one minimizes the orthogonal 


distances between the fitting function F and the data points. Important examples are fitting of a 


circle, an arc, or an ellipse to data points. 


Examples: 


1. Given f\(t) = 1 and f,(t) = t, find the least squares fit to the data points (1, 0.75), (2, 1.13), and 


(3, 1.39). We get the A, b, and xp from Example 1 in section 39.1, and 


F (Xo, t) = 0.45 + 0.32t. 
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If the third data point is changed to (3, 13.9), then the least squares solution changes to xp = 
(—7.890, 6.575)/, and the least squares fit becomes F (Xo, f) = —7.890 + 6.575t; this illustrates 
the sensitivity to outliers. 


39.3 Geometric and Algebraic Aspects 


Definitions: 


The columns of A € R”*” span the range of A, while the nullspace or kernel of A is the set of solutions 
to the homogeneous system Ax = 0: 


range(A) = {z = Ax|x € R"}, ker(A) = {x € R" | Ax = 0}. 


The four fundamental subspaces associated with Problem LS are range(A), ker(A‘), ker(A), and 
range(A‘). (See Section 2.4 for more information.) 


Facts: 


The first three facts can be found in [Str88, Sec. 2.4]; the remaining facts are discussed in [Bjo96, Chap. 1]. 
p denotes the rank of A: p = rank(A). 


1. If p =n < m, then the vector 0 of all zeros is the only element in ker(A). 

2. The spaces range(A) and ker(A‘) are subspaces of R” with dimensions p and m— p, respectively. 
The two spaces are orthogonal complements, ie,y!z = 0 for any pair (y € range(A), z € ker(A‘)), 
and range(A) @ ker(A’) = R”. 

3. The spaces ker(A) and range(A‘) are subspaces of R” with dimensions n—p and p, respectively. 
The two spaces are orthogonal complements. 

4. The least squares residual vector rp = b — Axy is an element in ker( A7). Combining this with the 
definition of r, we get the so-called augmented system associated with Problem LS: 


If p =n, then the augmented system is nonsingular and the solution components are rp and Xo. 

5. The vector Axo is the orthogonal projection of b onto range(A). 

6. The vector ro is the orthogonal projection of b onto ker(A7). 

7. If p <n, then the columns in A can be reordered such that A TT = [A Al , where IT is a 
permutation matrix and the submatrix A has p columns, range(A) = range(A). The permutation 
is not unique. 

8. The orthogonal projectors onto range( A) and ker(A) are given by AA! and I — A! A, respectively. 


Examples: 
1. The Figure 39.1 illustrates Facts 5 and 6 in the case m = 3,n = p = 2. 


b 


To 


FIGURE 39.1 
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2. For the problem in Example 1 in Section 39.1, both range(A) and ker(A™) are subspaces of R? 
given by, respectively, 


1 1 1 
range(A) =a |1] +6 2], ker(A‘) = y |—2], 
1 3 1 


with a, B,y € R. 
3. Fact 4 can be used to derive the normal equations: 


r=b—Axéker(A’) => A’(b— Ax) =0. 


39.4 Orthogonal Factorizations 


(See Section 5.5 and Section 38.4 for additional information on orthogonal factorizations.) 


Definitions: 


The real matrix Q is orthogonal if it is square and satisfies Q'Q=I. 
A QR factorization of a matrix A ¢ R”™*” with m > n has the form 


A=QR= a4] = OR, 
where Q € R”*” is orthogonal, R € R”™", Re R"" is upper triangular, and Q = Q.in. The form 
A = QR is the so-called “reduced” (or “skinny”) QR factorization. 
The singular value decomposition (SVD) of A € R”*" has the form 


A=UZV', 
where U € R”*” and V € R"*” are orthogonal matrices and & = diag(o),... ,o,) € R*" has diagonal 
elements 
0, >02>--->0,>0, r=min{m,n}. 


Letting uj; and v; denote the jth column in U and V, respectively, we can write 
a 
A= SS OjUjV; - 
j=l 


For k < r the matrix ee ojujv; is called the truncated SVD approximation to A. (See Sections 5.6, 
17, and 45 for more information about the singular value decomposition.) 


Facts: 
Except for Facts 1 and 8 see [Bjo96, Chap. 1]. Also see Section 39.5. 


1. [Str88, Chap. 3]. A QR factorization preserves rank: rank(R) = rank(R) = rank(A). 

2. AQR factorization is not unique, but two factorizations Q; R; and Q) Rp always satisfy R 1=D Re 
where D is diagonal with D;; = +1. 

3. The triangular factor. Rand the upper ‘triangular Cholesky factor C for the normal equations matrix 
A’ A always satisfy R = DC, where D is diagonal with Dj; = +1. 

4. If A has full rank and has the QR factorization A = QR, then xp can be found by back substitution 


in the upper triangular system Rx= Dy 
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5. If Q has not been saved, then we can use forward and back substitution to solve the seminormal 
equations: R™Rx = A’Db. For reasons of numerical stability, this must be followed by one step of 
iterative refinement; the complete process is called the corrected seminormal equations method. 

6. Let rank(A) = p <n < mand A= QR. The columns in Q.):) and Q.,p41:m are orthonormal 
bases of range(A) and ker( A’), respectively. 

7. Let A= UX V". Then p = rank(A) is equal to the number of strictly positive singular values: 
O, = +t+ S Op > 0, Opyr = +++ = Omin(mn) = 0. The columns in U,}.) and U.psiim are 
orthonormal bases of range( A) and ker(A‘), respectively, and the columns in V.:) and V;,p+1:n are 
orthonormal bases of range( A‘) and ker(A), respectively. 

8. [GV96, Chap. 12]. The TLS solution can be computed as follows: compute the SVD of the coefficient 
matrix A augmented with the right-hand side b, ie., [A, b] = US V7. If the smallest singular 
value 0,4 is simple, and if 6 = Veiag # 0, then 


-11y ~ TT VT ~ TT 
xT = —B Viennti> Er = —On+1 Unni Vieni Ty = —On+1 B Usn4i . 


Examples: 
1. For the problem from Example 1 in Section 39.1, we find [ A, b] = US V? with 


4.515 0.4248 0.3950 
E = diag |0.6198] , U,3 = |—0.8107], V3 = | 0.2796 
0.0429 0.4029 —0.8751 
Thus, 
0.4513 —0.0072 —0.0051 —0.015939 
T= 1531957 Er= | 0.0137 0.0097}, rr = | 0.030421 
: —0.0068 —0.0048 —0.015118 


Bt 
In Example 1 in section 39.1 we found xp = [0.4500 0.3200) . The difference between xp and x; 
is small because the problem is almost consistent and A is well conditioned; see Section 39.6 The 
elements in ry are about 80% of the elements in ro given in Example 1 in Section 39.1. 


39.5 Least Squares Algorithms 


Definitions: 


By least squares algorithms we mean algorithms for computing the least squares solution efficiently and 
stably on a computer. The algorithms should take into account the size of the matrix and, if applicable, 
also its structure. 


Facts: 
For real systems the following facts can be found in, e.g., [Bjo96], [Bjo04], and [LH95]. 


1. The algorithm which is least sensitive to the influence of rounding errors is based on the QR 
factorization of A: 


(a) Compute the reduced QR factorization A = OR. 


(b) Compute the vector 6 = Q'b (can be computed during the QR factorization algorithm 
without forming Q explicitly). 


(c) Compute x = R-!G via back substitution. 


The use of this algorithm was first suggested in [Gol65]. 
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2. If A is well conditioned, the normal equations can be used instead: 


(a) Compute the normal equation system M = A?A andd = A‘b. 


(b) Compute the Cholesky factorization M = C?C andy = C~1d (the vector y can be computed 
during the Cholesky algorithm). 


(c) Compute x = C~'y via back substitution. 


3. If Aor A7A is a Toeplitz matrix, use an algorithm that utilizes this structure to obtain the compu- 
tational complexity O(mn). 

4. If A is large and sparse, use a sparse QR factorization algorithm that avoids storing the matrix Q. 
If solving a system with the same A but a different right-hand side, use the corrected seminormal 
equations. 

5. Alternatively, if A is large and sparse, it may be preferable to use the augmented system approach 
because it may lead to less fill-in. Then a symmetric indefinite solver, such as the LDL’ factorization, 
must be used, cf. [GV96, Sec. 4.4]. 

6. If Ais large and the matrix-vector multiplications with A and A’ can be computed easily, then use 
the conjugate gradient algorithm on the normal equations. Several implementations are available; 
CGLS is the classical formulation; LSQR is more accurate for ill-conditioned matrices. 


39.6 Sensitivity 


Definitions: 


For A € R”*" with p = rank(A) < min(m,n) the condition number for the least squares problem is 
given by «(A) = o1/op, where 0; > 02 > --- > op > Oare the nonzero singular values of A. 

Let Xo and Xp denote the minimum-norm least squares solutions to the problems min ||b — Axl], 
and min \|b — Ax|| 2, the latter being a perturbation of the former. Define the quantities 


~ ~ re) 6 
84=|A-Alb, dp =|b—bp, = n=A=u(Ay—*. 
op Alla 


Facts: 
(See, e.g., [Bjo96, Sec. 1.4] or [LH95, Chap. 9].) 


1. If < 1, then the rank is not changed by the perturbation rank(A) = rank(A). 
2. If <1, then the relative perturbation in Xp, is bounded as 


Xom — Xomll2 E K(A) ( 5A 5p + nllFoll2 
Xomll2 ~~ 1L—n \WAll2 — | All2|xomll2 


If rank(A) = n, then the last term 7 is omitted. If the problem is consistent, i.e., r9 = 0, then the 
relative error can be expected to grow linearly with «. For ro 4 0 the contribution 7||r9||2 and the 
definition of n show that the relative error may grow as «( A)’. 

3. The condition number for the normal equations matrix is «(A7A) = «(A)*. Due to the finite 
computer precision, information may be lost when the normal equations are formed; see, e.g., 
[Bjo96, Sec. 2.2] and Example 2 below. 

4. Component-wise perturbation theory applies when component-wise perturbation bounds are avail- 
able for the errors in A and b; if 


)en to = b— Axo . 


JA— Al <elA| and [b—b| < (bl, 
where the absolute values and inequalities are interpreted componentwise, then 


[x — x| = €|A"| (|b] + [Al [xl) + (ATA) “| E7| [rl 
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and 


I= xllo Se {fA CAL xl + ID] +e fiCATA LAT He 


See [Bjo96, Secs. 1.4.5-6] for further references about component-wise perturbation analysis and 
a posteriori error estimation. 


Examples: 


1. We consider the problem from Example 1 in Section 39.1 and two perturbed versions of it, 


1 ol 0.75 of 0.8 1.1 = 0.79 
A= ]1 2], b= ]1.13], A= | 0.95 2 |, b= ]1.23 
1.3 1.39 1.1 2.95 1.30 


The matrix has full rank, so the least squares solution is unique (and equal to the minimum-norm 
least squares solution). The vectors 


_ {0.4500 ~ _ [0.5967 ~ _ [0.8935 
*0 = 10.3200}? ~~ Jo.2550]’ ~*~ 0.1280 
are the minimizers of ||b — Ax||2, \|b — Ax|l2, and \|b - Ax\l>, respectively. 


The matrix has condition number « (A) = 6.793, and7 = k(A)x|| A— All>/I| Allo = 0.4230 < 1. 
The relative errors and their upper bounds are 


a b—b 
Ia = olla _ 9 594 < K(A) [b= Ble _ 9 493 
\|Xoll2 || All2 [Xo lla 


Ixll2 | Alla Ilxoll2 


ae 1 b—b b—A 
I — oll _ gos c (n +4 Ib = bla + al st = 1.575. 


2. Consider the following matrix A and the corresponding normal equation matrix: 


1+6? 1 


1 1 
A=1|6 0 A'TA= 
: 1 146? 

0 6 zz 


If |5| < ./e (where ¢ is the machine precision), then the quantity 1 + 5? is represented by 1 on the 
computer and, therefore, the computed A’ A is singular. If we use Householder transformations 


wom he —-1l -l : . , 
to compute the QR factorization, we get R = | ec 3 A , so information about 6 is preserved. 


39.7 Up- and Downdating of QR Factorization 


Definitions: 
Given A € R™*" with m > n and its QR factorization, as well as a row vector a’ witha € R", updating 


of the factorization means computing the QR factorization of the augmented matrix =A=QR 


A 
al 


from the QR factors of A. Similarly, downdating means computing the QR factors of A from those of A. 
Up- and downdating algorithms require only O(mn) flops (compared to the O(mn7’) flops of recomputing 
the QR factors). 
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Facts: 
The following facts can be found in Chapter 3 of [Bjo96]. 


1. The matrix R can be updated to R without knowledge of Q by means a sequence of n Givens 
transformations G),..., Gy. 
2. The updating of Q to Q then takes the form 


Pp) Q 0 T T 
Q= kK i GT...GT. 


3. Downdating of RtoR requires the first row Of. of Q. Let G 1sasene's Gig be Givens rotations such 
that R = Gin-n +++ Gy ( R are ) is upper triangular. Then R = Rovesiiens 

4. If Q is not available, then its first row Q:; can be computed by the LINPACK/ Saunders algorithm 
or via hyperbolic rotations. If A is available, the corrected seminormal equations provide a more 
accurate algorithm. 

5. Up- and downdating algorithms are also available for the cases where a column is appended to or 
deleted from A. 

6. Up- and downdating of the Cholesky factor under a rank-one modification M = M+aa! is 
analytically equivalent to updating R from the QR factorization. In the downdating case the matrix 
M must be positive (semi)definite. 


Examples: 
1. Let 
1 1 0.5774 —0.7071 —0.4082 1.732 3.464 
A=]1 2] =QR= |0.5774 0 0.8165 0 1.414 
1 3 0.5774 0.7071 —0.4082 0 0 
Ifa? = (1 4), then 
1 1 2 5 
~ 1 2 a ine ‘ fend 0 2.236 
A= i 4|= QR with R= 0 0 
1 4 0 0 


The updated factor R is computed by augmenting R witha’ and applying two left Givens rotations 
G, and G;j to row pairs (1,4) and (2,4), respectively: 


1.732 3.464 % 4 er. 
R|_ | 0 1414) c. |o 1.414] @. |O 2.236 
gop @ 7 ee 0 0 
1 4 0 1.732 0 0 


39.8 Damped Least Squares 


Definitions: 

The damped least squares solution is the solution to the problem (AA + aI)x = A’b, where a > 0 
and A and bare real. Damped least squares is also known as ridge regression and Tikhonov (or Phillips) 
regularization. 
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Facts: 
(See, e.g., [Han98] for further details.) 


1. The two formulations 


A 
Jal 


min{||b — Ax||} + a||x||3} and min x 


2 


b 
al 
are analytically equivalent. 
2. The damping (controlled by the parameter w) reduces the variance of the solution, at the cost of 
introducing bias. 
3. If Cov(b) = I, then the covariance matrix for the damped least squares solution x, is 


Cov(xy) = (ATA+ a1) ATA(ATA +01) 


=VY?(x*+al)°V', 
where & and V are from the SVD of A. Hence, 
I|Cov(xa)|l2 = max{o7/(o7 + a)*} < (4a)", 


while ||Cov(xo) |l2 = ||(A7A) 1 lz = 0,-°, which can be much larger. 
4. The expected value of x,y is 


E(Xq) = (ATA+a1)'ATAX, 


which introduces a bias because E(x,) # E(x) whena > 0. 
5. The damped least squares problem can take the more general form 


min — Ax||5, + a|| Bx a +a x= 5 
in{||b — Ax||3 + a] Bx||3} (ATA +aB'B)x= A'b 


where || B - ||. defines a (semi)norm. The solution to this problem is unique when the nullspaces 
of A and B intersect trivially. 

6. [Han98, Chap. 7]. Some algorithms for computing @ are the discrepancy principle, generalized 
cross validation, and the L-curve criterion. 


Examples: 


1. Let 5 = 107° and consider the matrix and vectors 


1 1 0 

1 a 
A={1 146], Xo = if b= Ax, b=b+ |0 
1 14+26 6 


Obviously xo is the least squares solution to Ax + r = b with rp = 0. The minimizer of \|b — Ax|l2 
is ¥) = (0.5 1.5)", showing that for this problem the least squares solution is very sensitive to 
perturbations (the condition number is «(A) = 2.4-10°). Using a = 10~°, we obtain the damped 


least squares solutions 
_ |0.999995 su ~ _ {0.995 
* * | 1.000005 * = 11.005] ° 


Comparing the damped and the undamped least squares solutions X, and Xo to the perturbed 
least squares problem, we see that Xx is a better approximation to the unperturbed solution xo 
than Xo. 
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39.9 Rank Revealing Decompositions 


Definitions: 


A rank revealing decomposition is a two-sided orthogonal decomposition of the form 


A=URV' =U Vv"; 


0 


where U and V are orthogonal, and R is upper triangular and reveals the (numerical) rank of A in the 
size of its diagonal elements. 
The numerical rank k, of A, with respect to the threshold T, is defined as 


k, =minrank(A+ E)  subjectto ||E|l2 < t. 


Facts: 
(See, e.g., [Bjo96, Sec. 1.7.3-6], [Han98, Sec. 2.2], and [Ste98, Chap. 5].) 


1. [GV96, p. 73] The numerical rank k, is equal to the number of singular values greater than T, ie., 
OK, > T = Ok, 41- 

2. The singular value decomposition is rank revealing with the middle matrix R = X. The SVD is 
difficult to update. 

3. If Ais exactly rank deficient with rank(A) = p, then there always exists a pivoted QR factorization 
AN= QR with R of the form 


a Ruy Ra 
R —_ 
| 0 0 


| ; Ri € R’*?, rank(R) = rank(Ri1) = p, 


and a complete orthogonal decomposition of the form A = U RV", where 


= Riu 0 
0 0 


R= , Rye R?*?, — rank(R) = rank(Ry;) = p. 


The pseudoinverse of A is At = Vga. Uy: 

4. A basic solution can be computed from the pivoted QR factorization as xpn = I] Ree “ i: pb. The 
minimum-norm least squares solution is given in terms of the complete orthogonal decomposition 
as Xom = Vip Ri UL]. pb. 

5. The rank revealing QR (RRQR) decomposition is a pivoted QR factorization AT] = Q R such 
that 


oj/c; < o)( Risin) < oF < WRi:isills < cioi, eal Deere 


where o; is the ith singular value of A, 


c; = Vil(n — i) + min(i,n — i), 


and o;(Rj.;,1.,) denotes the smallest singular value of Rj.j,1.;. The RRQR factorization can be used 
to estimate the numerical rank k,. The RRQR factorization is not unique. 

6. The URV decomposition is a two-sided orthogonal decomposition A = UR V!, such that, for 
k=1,...,n, 


oe SO(Riiii) <Oj;, 1=1,...,k 
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and 
Oj < Oj-K(Retinktin) < G1 /Cy, i=k+1,...,n, 
where 
— (: WRertondct tll i 
oK(Ri:i,1:4)? = | Ritiink-tinllS 


There is also a ULV decomposition with a lower triangular middle matrix; both can be used to 
estimate the numerical rank of A. 

7. The RRQR, URV, and ULV decompositions can be updated, at slightly more cost than the QR 
factorization. 


Examples: 
1. The rank of A is revealed by the zero element in the (3,3) position of R: 


; 5 4 5.477 7.303 9.129 
A= oo a QR with R= 0 0.816 1.633 
45 ; 0 0 0 


Here the QR factorization is rank revealing (U = Q and V = I). 

2. Pivoting must be used to ensure that a QR factorization is rank revealing. The “standard column 
pivoting” often works well in connection with Householder transformations; here the pivot column 
in each stage is chosen to maximize the norm of the leading column of the submatrix yee to be 
reduced. Example: 


—4.4721 —2.2361 —2.6833 


AVX =HA= 0 0 0.8944 |, 
0 0 1 
—4.4721 —2.6833 —2.2361 100 
AUT = 0 0.8944 0 , T=]|0 0 II, 
0 1 0 0 10 


—4,4721 —2.6833 —2.2361 
R= A? = HAN = 0 —1.3416 0 
0 0 0 


3. The standard column pivoting strategy is not guaranteed to reveal the numerical rank; hence, the 
development of the RRQR and URV decompositions. 
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40.1 Introduction 


Let A be an n by n nonsingular matrix and b be an n-vector. As discussed in Chapter 38, Matrix 
Factorizations and Direct Solution of Linear Systems, the solution of the system of linear equations Ax = b 
using Gaussian elimination requires O(n*) operations, which typically include additions, subtractions, 
multiplications, and divisions. The solution also requires O(n”) words of storage. The computational 
complexity is based on the assumption that every element of the matrix has to be stored and operated on. 
However, linear systems that arise in many scientific and engineering applications can be large; that is, n 
can be large. It is not uncommon for n to be over hundreds of thousands or even millions. Fortunately, 
for these linear systems, it is often the case that most of the elements in the matrix A will be zero. 

Following is a simple example that illustrates where the zero elements come from. Consider the Laplace 
equation defined on a unit square: 


ax? + ay? >. 

Assume that u is known along the boundary. Suppose the square domain is discretized into a (k + 2) by 

(k + 2) mesh with evenly spaced mesh points, as shown in Figure 40.1. Also suppose that the mesh points 

are labeled from 0 tok + 1 in the x and y directions. For 0 < i < k +1, let the variables in the x direction 

be denoted by x;. Similarly, for0 < j < k +1, let the variables in the y direction be denoted by y;. 

The solution at (x;, y;) will be denoted by uj,; = u(x;, y;). To solve the Laplace equation numerically, the 

partial derivatives at (x;, y;) will be approximated, for example, using second-order centered difference 
approximations: 


2. 
Oru  Yin1,j ~ 2Uijg + Ui41,j 
2 x ; 
Ox? I(xi,y;) h2 

2 
Ou  4ij-1 — 2ui,; + Ui,j+1 
mes x , 
Oy? lxy;) h? 
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FIGURE 40.1 Discretization of a unit square. 


whereh = mH is the spacing between two mesh points in each direction. Here, it is assumed that ug, j, ux+1,;, 
Uio> Uik+1 are given by the boundary condition, for 1 < i,j < k. Using the difference approximations, 
the Laplace equation at each mesh point (x;, y;), 1 < i,j < k, is approximated by the following linear 


equation: 
Uj-1,j + Uij-1 — 4Ui,j + Mi4nj +Uij41 =9, forl <i,j <k. 


This leads to a system of k* by k? linear equations in k? unknowns. The solution to the linear system 
provides the approximate solution u;,;, 1 < i,j < k, at the mesh points. Note that each equation has 
at most five unknowns. Thus, the coefficient matrix of the linear system, which is ke? by k? and has k* 
elements, has at most 5k? nonzero elements. It is therefore crucial, for the purpose of efficiency (both in 
terms of operations and storage), to take advantage of the zero elements as much as possible when solving 
the linear system. The goal is to compute the solution without storing and operating on most of the zero 
elements of the matrix. This chapter will discuss some of techniques for exploiting the zero elements in 
Gaussian elimination. 

Throughout this chapter, the matrices are assumed to be real. However, most of the discussions are also 
applicable to complex matrices, with the exception of those on real symmetric positive definite matrices. 
The discussions related to real symmetric positive definite matrices are applicable to Hermitian positive 
definite matrices (which are complex but not symmetric). 


40.2 Sparse Matrices 


Definitions: 


A matrix A is sparse if substantial savings in either operations or storage can be achieved when the zero 
elements of A are exploited during the application of Gaussian elimination to A. 
The number of nonzero elments in a matrix A is denoted by nnz(A). 


Facts: [DER89], [GL81] 


1. Let A be an n by n sparse matrix. It often takes much less than n* words to store the nonzero 
elements in A. 

2. Let T be an n by n sparse triangular matrix. The number of operations required to solve the 
triangular system of linear equations Tx = b is O(nnz(T)). 
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Examples: 


1. A tridiagonal matrix T = [¢;,;] has the form 


bo 633 t34 


th—2n 3 th 2,n—2 than 1 


th 1,n—2 th—1n 1 ty 


tayn-1 tin 


where ¢;; #4 0 (1 <i <n), and tj4; 4 0, and t4,; 4 0,(1 < i < n—1). The matrix T is 
an example of a sparse matrix. If Gaussian elimination is applied to T with partial pivoting for 
numerical stability, tj 342, 1 <i < m—2,may become nonzero. Thus, in the worst case, there will be 
at most 5n nonzero elements in the triangular factorization (counting the 1s on the diagonal of the 
lower triangular factor). The number of operations required in Gaussian elimination is at most 5n. 

2. Typically only the nonzero elements of a sparse matrix have to be stored. One of the common ways 
to store a sparse matrix A is the compressed column storage (CCS) scheme. The nonzero elements are 
stored in an array (e.g., VAL) column by column, along with an integer array (e.g., IND) that stores 
the corresponding row subscripts. Another integer array (e.g., COLPTR) will be used to provide 
the index k where VAL(k) contains the first nonzero element in column k of A. Suppose that A is 
given below: 


ll 0O 16 19 22 
0 13 0 O- 23 
A=1/12 14 17 0 0 
0 O 18 20 24 
0 15 O 21 25 


The CCS scheme for storing the nonzero elements is depicted in Figure 40.2. For example, the 
nonzero elements in column 4 and the corresponding row subscripts can be found in VAL(s) and 
IND(s), where s = COLPTR(4), COLPTR(4) + 1, COLPTR(4) + 2, ---, COLPTR(5) — 1. Note 
that, for this example, COLPTR(6) = 16, which is one more than the number of nonzero elements 
in A. This is used to indicate the end of the set of nonzero elements. 


IND 14/3 )2)3)5 }143 }4}1)4)/5)1)2)4 45 
1/12 |13 |14 )15 |16 |17 19 |20 |21 |22 |23 |24 |25 
1|3 | 6 12 | 16 


FIGURE 40.2 An example of a compressed column storage scheme. 


VAL |1 18 


COLPTR 


9 
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3. The compressed row storage (CRS) scheme is another possibility of storing the nonzero elements of 
a sparse matrix. It is very similar to the CCS scheme, except that the nonzero elements are stored 
by rows. 

4. Let A = [a;,;] be an n by n sparse matrix. Let b = [b;] and c = [c;] be n-vectors. The following 
algorithm computes the product c = Ab, with the assumption that the nonzero elements of A are 
stored using the CRS scheme. 
fori = 1,2,---,ndo 

c; <0 
for s = ROWPTR(i), ROWPTR(z) + 1, ---, ROWPTR(s + 1) — 1 do 
qG—at VAL(s )binvis)- 

5. Let A = [aj,;] be an n by n sparse matrix. Let b = [b;] and c = [c;] be n-vectors. The following 
algorithm computes the product c = Ab, with the assumption that the nonzero elements of A are 
stored using the CCS scheme. 
fori = 1,2,---,ndo 

cj <0 
fori = 1,2,---,ndo 
for s = COLPTR(i), COLPTR(i) + 1,---, COLPTR(s + 1) — 1 do 
CIND(s) <— CinD(s) + VAL(s ) bj. 


40.3 Sparse Matrix Factorizations 


Mathematically, computing a triangular factorization of a sparse matrix using Gaussian elimination is 
really no different from that of a dense matrix. However, the two are very different algorithmically. Sparse 
triangular factorizations can be quite complicated because of the need to preserve the zero elements as 
much as possible. 


Definitions: 


Let A be a sparse matrix. An element of a sparse matrix A is a fill element if it is zero in A but becomes 
nonzero during Gaussian elimination. 
The sparsity structure of a matrix A = [a;,;] refers to the set Struct(A) = {(i, j) : aij A O}. 
Consider applying Gaussian elimination to a matrix A with row and column pivoting: 


A= P| MP) Mp--+ Pay Mn-1U Qn-1 +++ Q2Q1, 


where P; and Q; (1 <i < n — 1), are, respectively, the row and column permutations due to pivoting, 
M; (1 <i <n -— 1) isa Gauss transformation (see Chapter 38), and U is the upper triangular factor. Let 
L=M,+M,+---+M,-1 — (n— 2), where I is the identity matrix. Note that L is a lower triangular 
matrix. The matrix F = L + U — J is referred to as the fill matrix. 

The matrix A is said to have a zero-free diagonal if all the diagonal elements of A are nonzero. The 
zero-free diagonal is also known as a maximum transversal. 

No exact numerical cancellation between two numbers u and v means that u + v (or u — v) is nonzero 
regardless of the values of u and v. 


Facts: [DER89], [GL81] 


In the following discussion, A is a sparse nonsingular matrix and F is its fill matrix. 


1. [Duf81], [DW88] There exists a (row) permutation matrix P, such that P, A hasazero-free diagonal. 
Similarly, there exists a (column) permutation matrix P, such that AP, has a zero-free diagonal. 
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2. It is often true that there are more nonzero elements in F than in A. 

3. The sparsity structure of F depends on the sparsity structure of A, as well as the pivot sequence 
needed to maintain numerical stability in Gaussian elimination. This means that Struct(F) is 
known only during numerical factorization. As a result, the storage scheme cannot be created in 
advance to accommodate the fill elements that occur during Gaussian elimination. 

4. If A is symmetric and positive definite, then pivoting for numerical stability is not needed during 
Gaussian elimination. Assume that exact numerical cancellations do not occur during Gaussian 
elimination. Then Struct(A) C Struct(F). 

5. Let A be symmetric and positive definite. Assume that exact numerical cancellations do not occur 
during Gaussian elimination. Then Struct(F ) is determined solely by Struct(A). This implies that 
Struct(F ) can be computed before any numerical factorization proceeds. Knowing Struct(F) in 
advance allows a storage scheme to be set up prior to numerical factorization. 

6. [GN85] Suppose that A is a nonsymmetric matrix. Consider applying Gaussian elimination to a 
matrix A with partial pivoting: 


A= P| M,P)M)--+ Pry Mn-1U; 


where P; (1 < i < n— 1), isa row permutation due to pivoting, M; (1 <i < n— 1) isa Gauss 
transformation, and U is the upper triangular factor. Let F denote the corresponding fill matrix; 
that is, F = Mj + My +-+-+ M,_-; + U — (n — 1)I. The matrix product A’ A is symmetric 
and positive definite and, hence, has a Cholesky factorization ATA= LcLE. Assume that A has 
a zero-free diagonal. Then Struct(F) © Struct(Lc¢ + L?). This result holds for every legitimate 
sequence of pivots { P}, Po,--- , Py}. Thus, Struct(Lc) and Struct(L 7) can serve as upper bounds 
on Struct(L) and Struct(U), respectively. When A is irreducible, then Struct(L z) is a tight bound 
on Struct(U): for a given (i,j) € Struct(L 2), there is an assignment of numerical values to the 
nonzero elements of A so that U;,; is nonzero; this is referred to as an one-at-time result [GN93]. 
However, Struct(L) is not a tight bound on Struct(L). A tight bound on Struct(L) can be found 
in [GN87] and [GN93]. 


Examples: 


Some of the properties of sparse matrices, such as zero-free diagonals and reducible/irreducible matrices, 
depend only on the sparsity structures of the matrices; the actual values of the nonzero elements are 
irrelevant. As a result, for the examples illustrating such properties, only the sparsity structures will be 
taken into consideration. The numerical values will not be shown. Nonzero elements will be indicated by x. 


1. Following is an example of a reducible matrix: 


x x x x x x 
x x x x 
x x x 
x x x x x 
x x x 
A= 
x x x 
x x x x x x x 
x x x 
x x x x 


x 
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The matrix A can be put into a block upper triangular form using the following permutations: 


, = [2,4,7, 1,9, 6, 8,5, 3, 10], 


1, = [8, 6, 4,751, 3, 9, 5,2, 10]. 
Here, (i) = j means that row? of the permuted matrix comes from row j of the original matrix. 
Similarly, 2,(i) = j means that column i of the permuted matrix comes from column j of the 
original matix. When z, and 7 are applied to the identity matrix (separately), the corresponding 


permutation matrices are obtained: 


x x 


x x 


The permuted matrix P, AP, is shown below: 


x x x x 

x | xX x x | xX 
x x |X] xX x x | xX 

x x |X] xX x x 
x x x x 

PAP, = 

x x | xX 

x x x 

x x x 

x x x 
x 


The block triangular form has four 1 x 1 blocks, one 2 x 2 block, and one 4 x 4 block on the 
diagonal. 

2. The advantage of the block triangular form is that only the diagonal blocks have to be factored in 
the solution of a linear system. As an example, suppose that A is lower block triangular: 


Ail 
Az. Azz 


Ami Amz --- Amm 


Sparse Matrix Methods 40-7 


Here, m is the number of blocks on the diagonal of A. Consider the solution of the linear system 
Ax = b. Suppose that b and x are partitioned according to the block structure of A: 


b; Xx] 

b; p.6) 
b= and x= 

bi Xn 


Then the solution can be obtained using a block substitution scheme: 


k-1 
AkkXk = De — > Axpixi, for] <i<m. 


i=1 


Note that only the diagonal blocks Ay,, 1 < k < m have to be factored; Gaussian elimination does 
not have to be applied to the entire matrix A. This can result in substantial savings in storage and 
operations. 

3. In this example, the matrix A has zero elements on the diagonal: 


x x x x 
x x 
x x x x 

x x x x 
x x x x x x 

A= 

x x x x 
x x x x x 


x 
x 
x 
x 
x 
x 


The following permutation matrix P, when applied to the columns of A, will produce a matrix 
with a zero-free diagonal: 
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The sparsity structure of the permuted matrix is shown below: 


x x x x 
x x 
x x x x 
x x x x 
x x x x x x 
AP= 
x x x x 
x x x x x 
x x x x x x 
x x 
x x x x 


The permutation matrix P is obtained by applying the following permutation z to the columns of 
the identity matrix: 


m = [7,4, 8,5, 9, 2, 10, 3,6, 1]. 


Again, 7 (i) = j means that column of the permuted matrix comes from column j of the original 
matrix. 
4. Consider the following matrix: 


10 O 1 1 0 0 
1 10 0 0 1 0 0 
0 oO 10 0 0 1 0 
A=]0 0 1 10 0 0 14], 
0 1 0 0 10 0 41 
1 0 0 0 0 10 O 
0 0 0 0 1 0 10 


which is diagonal dominant. The matrix can be factored without pivoting for stability. The 
triangular factors are given below: 


1 10 0 1 0 1 0 0 
1 1 9 
0 1 10 a9 0 0 0 0 
0 1 10 0 0 0 
_ 1 = 1 
L= 0 0 1 v= 10 0 san 1 
1 1 991 1 
9 i to 9 1 Too + 1000 1 
1 1 10 99199 10 
10 0 100 0 991 1 9910 991 
100 1 981980 
0 0 0 0 991 99199 1 99199 


Note that A has 18 nonzero elements, whereas L + U has 26 nonzero elements, showing that there 
are more nonzero elements in the fill matrix than in A. 
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5. Consider the matrix A in the previous example. Suppose A = [4;, ;] is obtained from A by swapping 
the (1, 1) and (2, 1) elements: 


i @ 4 Lo 
10 10 TO: $0 
0 6? fo 0 1 0 
A={0 0 1 10 0 0 1 
0 1 +0 10 0 1 
1 0 0 0 10 0 
0 0 0 0 10 


When Gaussian elimination with partial pivoting is applied to A, rows 1 and 2 will be interchanged 
at step 1 of the elimination since |@| > |41,1|. It can be shown that no more interchanges are 
needed in the subsequent steps. The triangular factors are given by 


1 0 0 0 0 0 | 10 10 #O 1 0 
1 9 

TT 1 0 O 0 0 O 0 -1 1 mt 0 

0 0 1 0 0 0 a 0 0 10 O 0 0 

GS 105 0 ee 0 0 0o],U=|0 0 010 0 -¥ 1 

1 109 1 

0 -l1 i «(COO 1 0 0 0 0 O a0. 90 1 

1 1 10 10999 10 

10 1 10 0 109 1 0 0 0 0 0 1090 109 

10 10 108980 

0 0 0 0 109 ~~ 10999 1 0 0 0 0 0 10999 


Even though A (in the previous example) and A have the same sparsity structures, their fill matrices 
have different numbers of nonzero elements. For A, there are 27 nonzero elements in the fill matrix 
of A. While this example is small, it illustrates that the occurrence of fill elements in Gaussian 
elimination generally depends on both the sparsity structure and the values of the nonzero elements 
of the matrix. 

6. Consider the example in the last two examples again. Note that A has a zero-free diagonal. Both A 
and A have the same sparsity structure. The sparsity structure of A’ A is the same as that of A’ A: 


x x x x x 
x x 
x x x x 
ATA= x x 
x x x x x 
x x x 
x x XxX x x 


(Again, x denotes a nonzero element.) The Cholesky factor L¢ of the symmetric positive definite 
matrix A? A has the form: 


x 
x 
x x 
Lo= x x 
x x x 
x x x x 
x x x x 
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Note that Struct(L) 4 Struct(L), but Struct(L) C Struct(L¢) and Struct(£) C Struct(Lc). 
Similarly, Struct(U) 4 Struct(U), but Struct(U) C Struct(LZ) and Struct(U) C Struct(L 2). This 
example illustrates that when A has a zero-free diagonal the sparsity structure of the Cholesky 
factor of A’ A indeed contains the sparsity structure of L (or U"), irrespective of the choice of the 
pivot sequence. 

7. It is shown in an earlier example that fill elements do occur in Gaussian elimination of sparse 
matrices. In order to allow for these fill elements during Gaussian elimination, the simple storage 
schemes (CCS or CRS) may not be sufficient. More sophisticated storage schemes are often needed. 
The choice of a storage scheme depends on the choice of the factorization algorithm. There are 
many implementations of sparse Gaussian elimination, such as profile methods, left-looking meth- 
ods, right-looking methods, and frontal/multifrontal methods [DER89], [GL81]. Describing these 
implementations is beyond the scope of this chapter. Following is a list of pointers to some of the 
implementations. 


(a) Factorization of sparse symmetric positive definite matrices: [NP93], [RG91]. 


(b) Factorization of sparse symmetric indefinite matrices: [AGL98], [DR83], [DR95], [Duf04], 
[Liu87a], [Liu87b]. 


(c) Factorization of sparse nonsymmetric matrices: [DEG*99], [Duf77], [DR96], [GP88]. 


40.4 Modeling and Analyzing Fill 


A key component of sparse Gaussian elimination is the exploitation of the sparsity structure of the given 
matrix and its triangular factors. Graphs are useful in understanding and analyzing how fill elements are 
introduced in sparse Gaussian elimination. Some basic graph-theoretical tools and results are described 
in this section. Others can be found in [EL92], [GL81], [GL93], [GNP94], [Liu86], [Liu90], and [Sch82]. 

In this and the next sections, all graphs (bipartite graphs, directed graphs, and undirected graphs) are 
simple. That is, loops and multiple edges are not allowed. More information on graphs can be found in 
Chapter 28 (graphs), Chapter 29 (digraphs), and Chapter 30 (bipartite graphs). 


Definitions: 
Consider a sparse nonsymmetric matrix A = [a;,;]. Let R = {r1,12,--- Tn} be a set of “row” vertices 
associated with the rows of A. Similarly, let C = {c,,c2,--- ,c,} be a set of “column” vertices associated 


with the columns of A. The bipartite graph or bigraph of A, denoted by H(A) = (R,C, E), can be 
associated with the sparsity structure of A. There is an edge {r;,c;} € E ifand only a;,; 4 0. 

Let A = [a;,;] be a sparse nonsymmetric matrix. Suppose that A has a zero-free diagonal, and assume 
that the pivots are always chosen from the diagonal. Then the sparsity structure of A can be represented by 
a directed graph or digraph, denoted by (A) = (X, E). Here, X = {x,,x2,-++ ,X,}, with x, 1 <i <n, 
representing column i and row i of A. There is a directed edge or arc (x;,x;) € E if and only if a;,; 4 0, 
fori # j. (The nonzero elements on the diagonal of A are not represented.) 

Suppose A = [a;,;] is symmetric and positive definite, and assume that the pivots are always chosen 
from the diagonal. Then an undirected graph or graph (when the context is clear) G(A) = (X, E) can 
be used to represent the sparsity structure of A. Let X = {x),X2,--- ,X,}, with x; representing row i and 
column i of A. There is an (undirected) edge {x;, xj} € E ifand only ifa;,; 4 0 (and, hence, a;,; 4 0), for 
i # j. (The nonzero elements on the diagonal of A are not represented.) 

A path ina graph (which can bea bigraph, digraph, or undirected graph) is a sequence of distinct vertices 
(X5,>Xs.5°++ »Xs,) such that there is an edge between every pair of consecutive vertices. For bigraphs and 
undirected graphs, {x, p> XSpu1 },1 < p < t—1,isan (undirected) edge. For digraphs, the path is a directed 
path, and (Xsp>Xsp4i)> 1< p<t-—1,isanarc. 

Let A be an n by n matrix. After k (1 < k < n) steps of Gaussian elimination, the matrix remaining 
to be factored is the trailing submatrix that consists of the elements in the last (n — k) rows and the last 
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(n — k) columns of A. The graph (bipartite, digraph, or undirected graph) associated with this (n — k) by 
(n — k) trailing matrix is the k-th elimination graph. 


Facts: 
1. 


[PM83] Let H®, H, H®),..-, H™ be the sequence of elimination bigraphs associated with the 
Gaussian elimination of a sparse nonsymmetric matrix A = [a;,;]. The initial bigraph H) is the 
bigraph of A. Each elimination bigraph can be obtained from the previous one through a simple 
transformation. Suppose that the nonzero element a, , is chosen as the pivot at step k, 1 <k <n. 
Then the edge corresponding to a,,, {r;,¢;}, is removed from H (k-1) | together with all the edges 
incident to r, and c;. To obtain the next elimination bigraph H ©) from the modified H“-), an 
edge {r,c} is added if there is a path (r,c;,r;,c) in the original H“- and if {r,c} is not already in 
H‘*-), The new edges added to create H correspond to the fill elements introduced when a, ; 
is used to eliminate row s and column f. The bigraph H“?) represents the sparsity structure of the 
matrix remaining to be factored after row s and column ¢ are eliminated. 

[RT78] Let A be a sparse nonsymmetric matrix. Suppose that A has a zero-free diagonal and 
assume that pivots are restricted to the diagonal. Then Gaussian elimination can be modeled using 
elimination digraphs. Let rO, Pr), p@,..., 7 be the sequence of elimination digraphs. The 
initial digraph P is the digraph of A. Each elimination digraph can be obtained from the previous 
one through a simple transformation. Without loss of generality, assume that Gaussian elimination 
proceeds from row/column 1 to row/column n. Consider step k, 1 < k < n. Vertex x, is removed 
from P&-, together with all the arcs of the form (x;, u) and (v, x), where u and v are vertices in 
Tr&-). To obtain the next elimination digraph T) from the modified P*—, an arc (r, c) is added 
if there is a directed path (r, x,, c) in the original PD and if (r,c) is not already in PD. The 
new arcs added to create P) correspond to the fill elements introduced by Gaussian elimination 
at step k. The digraph P“) represents the sparsity structure of the matrix remaining to be factored 
after k steps of Gaussian elimination. 

[Ros72] For asymmetric positive definite matrix A, the elimination graphs associated with Gaussian 
elimination can be represented by (undirected) graphs. Let the elimination graphs be denoted by 
GO, GY, G,...,and G. The initial graph G© is the graph of A. Each elimination graph can 
be obtained from the previous one through a simple transformation. Without loss of generality, 
assume that Gaussian elimination proceeds from row/column 1 to row/column n. Consider step 
k,1<k <n. Vertex x; is removed from G*—)), together with all its incident edges. To obtain the 
next elimination graph G™ from the modified G“—"), an (undirected) edge {r, c} is added if there 
is a path (r, x;,c) in the original G-) and if {r, c} is not already in G*-))_ The new edges added 
to create G correspond to the fill elements introduced by Gaussian elimination at step k. The 
graph G“) represents the sparsity structure of the matrix remaining to be factored after k steps of 
Gaussian elimination. 

[GL80b], [RTL76] Consider an n by n symmetric positive definite matrix A. Let G = (X, E) be 
the (undirected) graph of A and let X = {x,,x2,--+ ,x,}. Denote the Cholesky factor of A by L. 
For i > j, the (i, j) element of L is nonzero if and only there is a path (xj, x5,,%5,,*++ »%s,.Xj) in 
G such that sp < j (<1), for 1 < p < t. Sucha path is sometimes referred to as a fill path. Note 
that f can be zero, which corresponds to {x;,x;} € E. 


Examples: 


1. 


As one may observe, the operations involved in the generation of the elimination bigraphs, digraphs, 
and graphs are very similar. Thus, only one example will be illustrated here. Consider the nonsym- 
metric matrix A = [q;,;] in Figure 40.3. The bigraph H of A is depicted in Figure 40.4. Suppose 
that a), is used to elimination row 1 and column 1. After the first step of Gaussian elimination, 
the remaining matrix is shown in Figure 40.5. Following the recipe given above, the edges {r1,c1}, 
{r1,¢3}, {r1, ce}, {r4, ci}, and {r7, c} are to be removed from the bigraph H in Figure 40.4. Then ry 
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x x x 
x 
x 
A=|x x x 
x x 
x x x 
x x 


FIGURE 40.3 A sparse nonsymmetric matrix A. 


ry} Cy 


Tr Co 
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T4 C4 
"5 C5 
"6 CGE 
r'7 C7 


FIGURE 40.4 The bigraph of the matrix A in Figure 40.3. 


x 
x 
ie + x x + 
x x 
x x x 
+ x + 


FIGURE 40.5 The remaining matrix after the first step of Gaussian elimination on the matrix A in Figure 40.3. 


and c, are also removed from the bigraph H. The new bigraph is obtained by adding to H the 
following edges: 


(a) {ra,c3} 
(b) {r4, co} 
(c) {r7,¢3} 
(d) {r7,co} 


The new bigraph is shown in Figure 40.6, in which the new edges are shown as dashed lines. Note 
that the new bigraph is exactly the bigraph of A’. 


because of the path (r4, cj, 11, ¢3) in the original H 
because of the path (r4, ci, 11, ¢¢) in the original H 
(17 
( 


because of the path (r7,c,,1),c3) in the original H 
because of the path (r7,c,,11,¢¢) in the original H 


Yeo ww 
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C2 


C3 


C4 


CE 


C7 


FIGURE 40.6 The bigraph of the matrix A’ in Figure 40.5. 


2. Let A = [a;,;] be a symmetric and positive definite matrix: 


x x x 
x x x 
x x x 
x x x 
A= 
x x x x 
x x x 
x x 
x x x 


Assume that exact numerical cancellations do not occur. Then the sparsity structure of the Cholesky 
factor L = [€;,;] of A is given below: 


x 
x 
x x x 
L= 
xX + x 
x x 
xX + + X 
x + se XX Eh OX 


The symbol + represents a fill element. The (undirected) graph G of A is shown in Figure 40.7. 
There are several fill paths in G. The fill path (xs, x2, x3, x1, xg) corresponds to the fill element £5 
in L. Another example is the fill path (x7, x4, x5, x6), which corresponds to the fill element 7, 
in L. 
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FIGURE 40.7 An example illustrating fill paths. 


40.5 Effect of Reorderings 


Let A bea sparse nonsingular matrix. As noted above, the occurrence of fill elements in Gaussian elimination 
generally depends on the values of the nonzero elements in A (which affect the choice of pivots ifnumerical 
stability is a concern) and the sparsity structure of A. This section will consider some techniques that will 
help preserve the sparsity structure in the factorization of A. 


Definitions: 


Let A bea sparse matrix. Suppose that G is a graph associated with A as described in the previous section; 
the graph can be a bipartite graph, directed graph, or undirected graph, depending on whether A is 
nonsymmetric or symmetric and whether the pivots are chosen to be on the diagonal. The fill graph G" of 
A is G, together with all the additional edges corresponding to the fill elements that occur during Gaussian 
elimination. 

An elimination ordering (or elimination sequence) for the rows (or columns) of a matrix is a bijection 
a: {1,2,---,n} — {1,2,--- ,n}. It specifies the order in which the rows (or columns) of the matrix are 
eliminated during Gaussian elimination. 

A perfect elimination ordering is an elimination ordering that does not produce any fill elements 
during Gaussian elimination. 

Consider an n by n sparse matrix A. Let f; and €; be the column indices of the first and last nonzero 
elements in row i of A, respectively. The envelope of A is the set 


{GZj):fi<j <4, forl <i <n}. 


That is, all the elements between the first and last nonzero elements of every row are in the envelope. 
The profile of a matrix is the number of elements in the envelope. 
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Facts: 


The problem of reordering a sparse matrix is combinatorial in nature. The facts stated below are some of 
fundamental ones. Others can be found, for example, in [GL81] and [Gol04]. 


i 


[GL81] An elimination ordering for the rows/columns of the matrix corresponds to a permutation 
of the rows/columns. 


. [DER89], [GL81] The choice of an elimination ordering will affect the number of fill elements in 


the triangular factors. 

[GL81], [Ros72] When A is sparse symmetric positive definite, an elimination ordering w can be 
determined by analyzing the sparsity structure of A. Equivalently, a can be obtained by analyzing 
the sequence of elimination graphs. The elimination ordering provides a symmetric permutation 
of A. Let P denote the permutation matrix corresponding to a. Let G* be the fill graph of the 
matrix P AP’. There exists a perfect elimination ordering for G* and G? isa chordal graph. 
Yan81] For sparse symmetric positive definite matrices, finding the optimal elimination ordering 
(i.e., an elimination ordering that minimizes the number of fill elements in the Cholesky factor) is 
NP-complete. This implies that almost all reordering techniques are heuristic in nature. 

DER89] When A is a sparse nonsymmetric matrix, the elimination orderings for the rows and 
columns of A have to be chosen to preserve the sparsity structure and to maintain numerical 
stability. 

GN85] Suppose that A is a sparse nonsymmetric matrix. If partial pivoting (by rows) is used to 
maintain numerical stability, then an elimination ordering for the columns of A can be chosen to 
preserve the sparsity structure. 


Examples: 


1. 


Consider the following two diagonally dominant matrices: 


7 1 -1 1 -i1 7 0 0 0 1 

-1 7 0 0 0 0 7 0 0 -l 

A= 1 0 7 0 0 and B= 0 0 7 0 1 
—1 0 0 7 0 0 0 0 7 —-1l 

1 0 0 0 7 -1 1 -1 1 7 


Applying Gaussian elimination to A produces the following triangular factors: 


1 Oe. SO OO ae eat ti <1 

1 50 1 1 1 

= 1 1 357 7 7 
A=| 5 3% 0 0} |0 0 3 i 
1 1 1 364 Zz 

7 5% OTT 1 0) j0 0 Sr OT BT 

1 1 1 1 371 


i) 07 0s 0) Ve Os Oe OF 
Oa, OS O20 POR 98-Oh S 
B=/| 00 100/]0070 1 
Oe Oo. Tee 0) OOo 7 St 
Ved AN a 53 
—5 7 77 7 WO O00 F 


The two matrices A and B have the same numbers of nonzero elements, but their respective 
triangular factors have very different numbers of fill elements. In fact, the triangular factors of B 
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2. 
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have no fill elements. Note that B can be obtained by permuting the rows and columns of A. Let P 
be the following permutation matrix: 


~ 

II 
HK ooco 
oroo9oo 
oorC Oo 


0 
1 
0 
0 
0 


oocUCcOODlUCOOUlUMr 


Then B = PAP?!; that is, B is obtained by reversing the order of the rows and columns. This 
example illustrates that permuting the rows and columns of a sparse matrix may have a drastic 
effect on the sparsity structures of the triangular factors in Gaussian elimination. 

A popular way to preserve the sparsity structure of a sparse matrix during Gaussian elimination 
is to find elimination orderings so that the nonzero elements in the permuted matrix are near 
the diagonal. This can be accomplished, for example, by permuting the matrix so that it has a 
small envelope or profile. The Cuthill-McKee algorithm [CM69] and the reverse Cuthill-McKee 
algorithms [Geo71], [LS76] are well-known heuristics for producing reorderings that reduce the 
profile of a sparse symmetric matrix. The permuted matrix can be factored using the envelope 
or profile method [GL81], which is similar to the factorization methods for band matrices. The 
storage scheme for the profile method is very simple. The permuted matrix is stored by rows. If the 
symmetric matrix is positive definite, then for every row of the permuted matrix, all the elements 
between the first nonzero element and the diagonal elements are stored. It is easy to show that the 
fill elements in the triangular factor can only occur inside the envelope of the lower triangular part 
of the permuted matrix. 

Although the profile method is easy to implement, it is not designed to reduce the number of fill 
elements in Gaussian elimination. The nested dissection algorithm [Geo73], which is based on a 
divide-and-conquer idea, is a well-known heuristic for preserving the sparsity structure. Let A be 
a symmetric positive definite matrix, and let G = (X, E) be the (undirected) graph of A. Without 
loss of generality, assume that G is connected. Let S C X. Suppose that S is removed from G. Also 
assume that all edges incident to every vertex in S are removed from G. Denote the remaining graph 
by G(X — S). If S is chosen so that G(X — S) contains one or more disconnected components, 
then the set S is referred to as a separator. Consider the following reordering strategy: renumber the 
vertices of the disconnected components of G(X — S) first, and renumber of the vertices of S last. 
Now pick vertex x in one component and vertex y in another component. The renumbering scheme 
ensures that there is no fill path between x and y in G. This is a heuristic way to limit the creation of 
fill elements. The renumbering corresponds to a symmetric permutation of the rows and columns of 
the matrix. Consider the example in Figure 40.8. The removal of the vertices in S (together with the 
incident edges) divides the mesh into two joint meshes (labeled G; and G2). Suppose that the mesh 
points in G; are renumbered before those in G2. Then the matrix on the right in Figure 40.8 shows 
the sparsity structure of the permuted matrix. Note the block structure of the permuted matrix. The 
blocks A; and A, correspond to mesh points in G; and G), respectively. The block As corresponds 
to the mesh points in S. The nonzero elements in the off-diagonal blocks C,/C2 correspond to 
the edges between G,/G, and S. The unlabeled blocks are entirely zero. This is referred to as the 
“dissection” strategy. When the strategy is applied recursively to the disconnected components, 
more zero (but smaller) blocks will be created in A; and A2. The resulting reordering is called a 
“nested dissection” ordering. It can be shown that, when the nested dissection algorithm is applied 
to a k by k mesh (like the example in the Introduction), the number of nonzero elements in the 
Cholesky factor will be O(k? log k) and the number of operations required to compute the Cholesky 
factor will be O(k?). Incidentally, for the k by k mesh, it has been proved that the number of nonzero 
elements in the Cholesky factor and the number of operations required to compute the triangular 
factorization are at least O(k? logk) and O(k*), respectively [HMR73], [Geo73]. Thus, nested 
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FIGURE 40.8 An example of the dissection strategy. 


dissection orderings can be optimal asymptotically. In recent years, higher quality nested dissection 
orderings have been obtained for general sparse symmetric matrices by using more sophisticated 
graph partitioning techniques to generate the separators [HR98], [PSL90], [Sch01]. 

4. The nested dissection algorithm is a “top-down” algorithm since it identifies the vertices 
(i.e., rows/columns) to be reordered last. The minimum degree algorithm [TW67] isa “bottom-up” 
algorithm. It is a heuristic that is best described using the elimination graphs. Let G be the (undi- 
rected) graph ofa sparse symmetric positive definite matrix. The minimum degree algorithm picks 
the vertex x, with the smallest degree (i.e., the smallest number of incident edges) to be eliminated; 
that is, x,, is to be reordered as the first vertex. Then x,,, together with all the edges incident to x,,, 
are eliminated from G to generate the next elimination graph G"). This process is repeated until 
all the vertices are eliminated. The order in which the vertices are eliminated is a minimum degree 
ordering. Note that if several vertices have the minimum degree in the current elimination graph, 
then ties have to be broken. It is well known that the quality of a minimum degree ordering can be 
influenced by the choice of the tie-breaking strategy [BS90]. Several efficient implementations of 
the minimum degree algorithm are available [ADD96], [GL80b], [GL80a], [Liu85]. An excellent 
survey of the minimum degree algorithm can be found in [GL89]. 

5. The minimum deficiency algorithm [TW67] is another bottom-up strategy. It is similar to the 
minimum degree algorithm, except that the vertex whose elimination would introduce the fewest 
fill elements will be eliminated at each step. In general, the minimum deficiency algorithm is 
much more expensive to implement than the minimum degree algorithm. This is because the 
former one needs the look-ahead to predict the number of fill elements that would be introduced. 
However, inexpensive approximations to the minimum deficiency algorithms have been proposed 
recently [NR99], [RE98]. 

6. For a sparse nonsymmetric matrix A = [aj,;], there are analogs of the minimum degree and 
minimum deficiency algorithms. The Markowitz scheme [Mar57] is the nonsymmetric version of 
the minimum degree algorithm for sparse nonsymmetric matrices. Recall that nnz(Aj,1.,) is the 
number of nonzero elements in rowi of A and nnz(A;.,,;) is the number of nonzero elements 
in column j of A. For each nonzero a;,; in A, define its “Markowitz” cost to be the product 
[nnz(Ajan) — 1] [nna(Aj-n,;) — 1], which would be the number of nonzero elements in the rank-1 
update if a;,; were chosen as pivot. At each step of Gaussian elimination, the nonzero element that 
has the smallest Markowitz cost will be chosen as the pivot. After the elimination, the Markowitz 
costs of all the nonzero elements, including the fill elements, are updated to reflect the change in 
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the sparsity structure before proceeding to the next step. If A is symmetric and positive definite, 
and if pivots are chosen from the diagonal, then the Markowitz scheme is the same as the minimum 
degree algorithm. 

7. The Markowitz scheme for sparse nonsymmetric matrices attempts to preserve the sparsity struc- 
ture by minimizing the number of nonzero elements introduced into the triangular factors at each 
step of Gaussian elimination. The resulting pivots may not lead to a numerically stable factorization. 
For example, the magnitude of a pivot may be too small compared to the magnitudes of the other 
nonzero elements in the matrix. To enhance numerical stability, a modified Markowitz scheme 
is often used. Denote the matrix to be factored by A = [a,j]. Lets = max{laj,j| : ai; 4 O}. 
Let t be a given tolerance; e.g., t can be 0.01 or 0.001. Without loss of generality, consider the 
first step of Gaussian elimination of A. Instead of considering all nonzero elements in A, let 
C = {a;,j : \ai,;| = ts}. Thus, C is the subset of nonzero elements whose magnitudes are larger 
than or equal to T s; it is the set of candidate pivots. Then the pivot search is limited to applying the 
Markowitz scheme to the nonzero elements in C. This is a compromise between preserving spar- 
sity and maintaining numerical stability. The two parameters t and s are usually fixed throughout 
the entire Gaussian elimination process. The modified scheme is often referred to as the threshold 
pivoting scheme [Duf77]. 

8. As noted earlier, if Gaussian elimination with partial pivoting is used to factor a sparse non- 
symmetric matrix A, then the columns may be permuted to preserve sparsity. Suppose that 
A has a zero-free diagonal, and let Lc be the Cholesky factor of the symmetric positive defi- 
nite matrix A’ A. Since Struct(L) C Struct(Lc) and Struct(U) C Struct(LZ ), a possibility is 
to make sure that Lc is sparse. In other words, one can choose a permutation P,. for A’ A 
to reduce the number of nonzero elements in the Cholesky factor of Pe (A™ A) P... Note that 
Pi (AT A)Pre = (AP,.)'(AP_.)- Thus, P;. can be applied to the columns of A [GN85], 
[GN87]. 
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Given an n by n nonsingular matrix A and an n-vector b, the linear system Ax = b can always be solved for 
x by Gaussian elimination. The work required is approximately 21? /3 operations (additions, subtractions, 
multiplications, and divisions), and, in general, n? words of storage are required. This is often acceptable 
if n is of moderate size, say n < 1000, but for much larger values of n, say, n © 10°, both the work and 
storage for Gaussian elimination may become prohibitive. 

Where do such large linear systems arise? They may occur in many different areas, but one impor- 
tant source is the numerical solution of partial differential equations (PDEs). Solutions to PDEs can be 
approximated by replacing derivatives by finite difference quotients. For example, to solve the equation 


a a a a a a 
oo (ax 215) + ay (aty.25") + ae (ax 215") = f(x,y,z) inQ 


u=g_ onboundary of Q, 


on a three-dimensional region Q, where a, f, and g are given functions with a bounded away from 0, one 
might first divide the region Q into small subregions of width h in each direction, and then replace each 
partial derivative by a centered difference approximation; e.g., 


1 
— (a5) (x,y,z) © plate + h/2, y,z)(u(x +h, y,z) — u(x, y,z)) 
—a(x —h/2, y,z)(u(x, y,z) — u(x —h, y,z))], 
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with similar approximations for 0/dy(adu/dy) and 0/dz(adu/dz). If the resulting finite difference ap- 
proximation to the differential operator is set equal to the right-hand side value f(x;, yj,z,) at each of 
the interior mesh points (x;, yj,zx),i = 1,...,m, j =1,...,m,k =1,... , m3, then this gives a system 
of n = njnzn3 linear equations for the n unknown values of u at these mesh points. If u;;, denotes the 
approximation to u(x;, yj,2Zx), then the equations are 


1 
ye lai + h/2, yj, Zk) (Uis1jk — Wijk) — a(x; — h/2, yj, 2K) (uijk — Ui-1,),k) 


+ a(xj, yj +h/2,Z)(Uijaik — Uijk) — a(x, Yj — h/2, ZK) (ijk — Ui,j-1k) 
+ A(Xis Vj» Zk +h/2)(ui,jkea — Uijk) — A(Xis YjsZk — h/2)(uijk — Ui,jk-1)] 
= f (Xi, Vj. Zk). 


The formula must be modified near the boundary of the region, where known boundary values are added 
to the right-hand side. Still the result is a system of linear equations for the unknown interior values of u. 
If n, = ny = n3 = 100, then the number of equations and unknowns is 100° = 10°. 

Notice, however, that the system of linear equations is sparse; each equation involves only a few 
(in this case seven) of the unknowns. The actual form of the system matrix A depends on the numbering of 
equations and unknowns. Using the natural ordering, equations and unknowns are ordered first by 7, then 
j» then k. The result is a banded matrix, whose bandwidth is approximately 1,2, since unknowns in any 
z plane couple only to those in the same and adjacent z planes. This results in some savings for Gaussian 
elimination. Only entries inside the band need be stored because these are the only ones that fill in (become 
nonzero, even if originally they were zero) during the process. The resulting work is about 2(nyn2)?n op- 
erations, and the storage required is about n,n2n words. Still, this is too much when ny = n2 = n3 = 100. 
Different orderings can be used to further reduce fill in, but another option is to use iterative methods. 

Because the matrix is so sparse, matrix-vector multiplication is very cheap. In the above example, 
the product of the matrix with a given vector can be accomplished with just 7n multiplications and 6n 
additions. The nonzeros of the matrix occupy only 7n words and, in this case, they are so simple that 
they hardly need be stored at all. If the linear system Ax = b could be solved iteratively, using only 
matrix-vector multiplication and, perhaps, solution of some much simpler linear systems such as diagonal 
or sparse triangular systems, then a tremendous savings might be achieved in both work and storage. 

This section describes how to solve such systems iteratively. While iterative methods are appropriate 
for sparse systems like the one above, they also may be useful for structured systems. If matrix-vector 
multiplication can be performed rapidly, and if the structure of the matrix is such that it is not necessary to 
store the entire matrix but only certain parts or values in order to carry out the matrix-vector multiplication, 
then iterative methods may be faster and require less storage than Gaussian elimination or other methods 
for solving Ax = b. 


41.1 Krylov Subspaces and Preconditioners 


Definitions: 


An iterative method for solving a linear system Ax = b is an algorithm that starts with an initial guess 
Xo for the solution and successively modifies that guess in an attempt to obtain improved approximate 
solutions xj, X2,.... 

The residual at step k of an iterative method for solving Ax = b is the vector r, = b — Axx, where xx 
is the approximate solution generated at step k. The initial residual is ry = b — Axo, where Xp is the initial 
guess for the solution. 

The error at step k is the difference between the true solution A~'b and the approximate solution xx: 
QA = A'b— Xk. 

A Krylov space is a space of the form span{q, Aq, A’q, ... , AX~!q}, where A is an n by n matrix and q 
is an n-vector. This space will be denoted as K;.(A, q). 
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A preconditioner is a matrix M designed to improve the performance of an iterative method for solving 
the linear system Ax = b. Linear systems with coefficient matrix M should be easier to solve than the 
original linear system, since such systems will be solved at each iteration. 

The matrix M~'A (for left preconditioning) or AM! (for right preconditioning) or L~!AL~* 
(for Hermitian preconditioning, when M = LL*) is sometimes referred to as the preconditioned 
iteration matrix. 

Another name for a preconditioner is a splitting; that is, if A is written in the form A = M — N, then 
this is referred to as a splitting of A, and iterative methods based on this splitting are equivalent to methods 
using M as a preconditioner. 

A regular splitting is one for which M is nonsingular with M~' > 0 (elementwise) and M > A 
(elementwise). 


Facts: 


The following facts and general information on Krylov spaces and precondtioners can be found, for 
example, in [Axe95], [Gre97], [Hac94], [Saa03], and [Vor03]. 


1. Aniterative method may obtain the exact solution at some stage (in which case it might be considered 
a direct method), but it may still be thought of as an iterative method because the user is interested 
in obtaining a good approximate solution before the exact solution is reached. 

2. Each iteration of an iterative method usually requires one or more matrix-vector multiplications, 
using the matrix A and possibly its Hermitian transpose A*. An iteration may also require the 
solution of a preconditioning system Mz = r. 

3. The residual and error vector at step k of an iterative method are related by r, = Aex. 

4. All of the iterative methods to be described in this chapter generate approximate solutions x,, k = 
1,2,..., such that x, — Xo lies in the Krylov space span{zo, Cz,... 5 Ck-1z}, where Zp is the initial 
residual, possibly multiplied by a preconditioner, and C is the preconditioned iteration matrix. 

5. The Jacobi, Gauss-Seidel, and SOR (successive overrelaxation) methods use the simple iteration 


Xk = Xe + M(b— Axi), k=1,2,..., 


with different preconditioners M. For the Jacobi method, M is taken to be the diagonal of A, while 
for the Gauss-Seidel method, M is the lower triangle of A. For the SOR method, M is of the form 
w !D— L, where D is the diagonal of A, —L is the strict lower triangle of A, and w is a relaxation 
parameter. Subtracting each side of this equation from the true solution A~!b, we find that the 
error at step k is 


e, = (I —M'A)a_; =... = (1 — M7! A)‘ey. 


Subtracting each side of this equation from eg, we find that x, satisfies 
€y — ek = Xk — X= [1 — (I — M'A)* Jeo 


‘ (k 
= a) (—1))-'(M71A)J"| 2, 


aa 


where Z) = M7! Aeg = M7'ro. Thus, xx — Xq lies in the Krylov space 
span{Zo, (M7! A)zo,... (M~! A)¥-1z9}. 


6. Standard multigrid methods for solving linear systems arising from partial differential equations are 
also of the form x, = X,_1 + M~'1_1. For these methods, computing M~'r,_ involves restricting 
the residual to a coarser grid or grids, solving (or iterating) with the linear system on those grids, 
and then prolonging the solution back to the finest grid. 
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FIGURE 41.1 Convergence of iterative methods for the problem given in the introduction with a(x, y,z) =1+x+ 
3yz, h = 1/50. Jacobi (dashed), Gauss-Seidel (dash-dot), and SOR with w = 1.9 (solid). 


Applications: 


1. Figure 41.1 shows the convergence of the Jacobi, Gauss-Seidel, and SOR (with w = 1.9) iterative 
methods for the problem described at the beginning of this chapter, using a mildly varying coefficient 
a(x, y,z) = 1+x-+3yzon the unit cube Q = [0, 1] x [0, 1] x [0, 1] with homogeneous Dirichlet 
boundary conditions, u = 0 on dQ. The right-hand side function f was chosen so that the solution 
to the differential equation would be u(x, y,z) = x(1 — x)y?(1 — y)z(1 — z)*. The region was 
discretized using a 50 x 50 x 50 mesh, and the natural ordering of nodes was used, along with a 
zero initial guess. 


41.2 Optimal Krylov Space Methods for Hermitian Problems 


Throughout this section, we let A and b denote the already preconditioned matrix and right-hand side 
vector, and we assume that A is Hermitian. Note that if the original coefficient matrix is Hermitian, then 
this requires Hermitian positive definite preconditioning (preconditioner of the form M = LL* and 
preconditioned matrix of the form L~'AL~*) in order to maintain this property. 


Definitions: 


The Minimal Residual (MINRES) algorithm generates, at each step k, the approximation x, with x, — x9 € 


K;(A, 19) for which the 2-norm of the residual, ||r,|| = (ty, 1)!/, is minimal. 


The Conjugate Gradient (CG) algorithm for Hermitian positive definite matrices generates, at each step 
k, the approximation x, with x, —xo € Kx(A, ro) for which the A-norm of the error, ||ex|| 4 = (ex, Aex) 1/2. 
is minimal. (Note that this is sometimes referred to as the A!/2-norm of the error, e.g., in Chapter 37 of 
this book.) 


The Lanczos algorithm for Hermitian matrices is a short recurrence for constructing an orthonormal 
basis for a Krylov space. 


Iterative Solution Methods for Linear Systems 41-5 


Facts: 


The following facts can be found in any of the general references [Axe95], [Gre97], [Hac94], [Saa03], and 
[Vor03]. 


1. The Lanczos algorithm [Lan50] is implemented as follows: 


Lanczos Algorithm. (For Hermitian matrices A) 
Given qy with ||qi|| = 1, set Bo = 0. For j = 1,2,..., 


Gj41 = Agj — Bj-iqj-1. Set aj = (Gj41,4j), Gjaa <— Giri — qj. 


Bj = llGjsill, jaa = Gje1/8;- 


2. It can be shown by induction that the Lanczos vectors qi, q2,... produced by the above algorithm 
are orthogonal. Gathering the first k vectors together as the columns of an n by k matrix Q,, this 
recurrence can be written succinctly in the form 


AQk = QeTe + Brdusiék' 


where & = (0,... ,0,1)7 is the kth unit vector and J is the tridiagonal matrix of recurrence 
coefficients: 
a By 
By 
k= 
Br-1 
Ber 


The above equation is sometimes written in the form 


AQk = Quai Ty, 


where T;, is the k + 1 by k matrix whose top k by k block is 7; and whose bottom row is zero except 
for the last entry which is By. 

3. Ifthe initial vector q; in the Lanczos algorithm is taken to be q, = r/||ro||, then the columns of Q; 
span the Krylov space K;(A, ro). Both the MINRES and CG algorithms take the approximation x, 
to be of the form x9 + Q;y, for a certain vector yx. For the MINRES algorithm, y, is the solution 
of the k + 1 by k least squares problem 


a B& — Tryll, 


where 6 = |lro|| and &, = (1,0,... ,0)? is the first unit vector. For the CG algorithm, y, is the 
solution of the k by k tridiagonal system 


Thy = Bé. 
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4. The following algorithms are standard implementations of the CG and MINRES methods. 


Conjugate Gradient Method (CG). 
(For Hermitian Positive Definite Problems) 


Given an initial guess x9, compute f9 = b— Axy and setpp=ro. Fork =1,2,..., 


Compute Ap,_1. 


= — _entk-1) 
Set X_ = Xk-1 + Gp-1Pk-1, Where az_| = ay ; 


Compute rp = rh—1 — Ap—1 APx-1. 


(Tkok) 
(fk-15Tk-1) * 


Set px = tq + Dp px—1, where bp) = 


Minimal Residual Algorithm (MINRES). (For Hermitian Problems) 


Given x, compute ro = b — Axo and set qi = ro/|lFoll- 
Initialize € = (1,0,... ,0)7, B = ||rol|. Fork = 1,2,..., 


Compute qx41,0% = T(k,k),and By = T(k+1,k) = T(k,k+1) using the Lanczos algorithm. 


Apply rotations F,_2 and Fx_, to the last column of T; that is, 


T(k — 2, k) Ck-2 = Sk—2 0 : 
Ge a = & ae aah ae 
T(k —1,k) ego Sea Vel DRA 
( T(k,k) )- ie sa | T(k,k) ie oe 


Compute the k" rotation, cz and s;, to annihilate the (k + 1,k) entry of T: 
ce = |T(kk)|//IT(K RK)? + IT + LK), 5¢ = ceT(k + 1,k)/T(k,k). 


Apply k‘” rotation to & and to last column of T: 
&(k) ck sk\ [ &(k) 
Ge 4 Pe & ( 0 : 
T(k,k) <— cy T(k,k) + 5.T(k+1,k), T(k+1,k) — 0. 


Compute px_-1 = [qx — T(k — 1,k)px—-2 — T(k — 2,k)pe_3]/T(k, k), where undefined terms 
are zero for k < 2. 


Set X_ = Xk-1 + Ap—1Pk—1, where az_; = BE(k). 


5. In exact arithmetic, both the CG and the MINRES algorithms obtain the exact solution in at most 
n steps, since the affine space x9 + K,,(A, ro) contains the true solution. 
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FIGURE 41.2 Convergence of MINRES (solid) and CG (dashed) for the problem given in the introduction with 
a(x,y,z) =1+x+3yz,h = 1/50. 


Applications: 


1. Figure 41.2 shows the convergence (in terms of the 2-norm of the residual) of the (unprecondi- 
tioned) CG and MINRES algorithms for the same problem used in the previous section. 
Note that the 2-norm of the residual decreases monotonically in the MINRES algorithm, but 
not in the CG algorithm. Had we instead plotted the A-norm of the error, then the CG convergence 
curve would have been below that for MINRES. 


41.3 Optimal and Nonoptimal Krylov Space Methods 
for Non-Hermitian Problems 


In this section, we again let A and b denote the already preconditioned matrix and right-hand side vector. 
The matrix A is assumed to be a general nonsingular n by n matrix. 


Definitions: 


The Generalized Minimal Residual (GMRES) algorithm generates, at each step k, the approximation x, 
with x, — Xo € Kx(A,1r9) for which the 2-norm of the residual is minimal. 

The Full Orthogonalization Method (FOM) generates, at each step k, the approximation x, with 
Xx — Xo € K;(A, r9) for which the residual is orthogonal to the Krylov space K;.(A, ro). 

The Arnoldi algorithm is a method for constructing an orthonormal basis for a Krylov space that 
requires saving all of the basis vectors and orthogonalizing against them at each step. 

The restarted GMRES algorithm, GMRES(j), is defined by simply restarting GMRES every j steps, 
using the latest iterate as the initial guess for the next GMRES cycle. Sometimes partial information from 
the previous GMRES cycle is retained and used after the restart. 

The non-Hermitian (or two-sided) Lanczos algorithm uses a pair of three-term recurrences involving 
A and A* to construct biorthogonal bases for the Krylov spaces Kx(A, 19) and K;(A*, fo), where fo is a 
given vector with (ro, fo) # 0. If the vectors vj,... , Vv, are the basis vectors for K;,(A,19), and wi,... , Wk 
are the basis vectors for K,(A*, fo), then (vj, wj) = 0 fori  j. 

In the BiCG (biconjugate gradient) method, the approximate solution x, is chosen so that the residual 
ty is orthogonal to K;,(A*, fo). 
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In the QMR (quasi-minimal residual) algorithm, the approximate solution x, is chosen to minimize a 
quantity that is related to (but not necessarily equal to) the residual norm. 

The CGS (conjugate gradient squared) algorithm constructs an approximate solution x, for which 
k= g7(A)ro, where ¢;(A) is the kth degree polynomial constructed in the BiCG algorithm; that is, the 
BiCG residual at step k is yx (A)ro. 

The BiCGSTAB algorithm combines CGS with a one or more step residual norm minimizing method 
to smooth out the convergence. 


Facts: 
1. The Arnoldi algorithm [Arn51] is implemented as follows: 


Arnoldi Algorithm. 
Given qy with ||qy|| = 1. For j = 1,2,..., 
Gj1 = Aqj. Fori =1,...5 9, hij = (Gj+1.4i)> G41 <— Gi — hijai- 


hjasj = Weill, jer = Gj41/ 2 j41;- 


2. Unlike the Hermitian case, if A is non-Hermitian then there is no known algorithm for finding the 
optimal approximations from successive Krylov spaces, while performing only O(n) operations 
per iteration. In fact, a theorem due to Faber and Manteuffel [FM84] shows that for most non- 
Hermitian matrices A there is no short recurrence that generates these optimal approximations for 
successive values k = 1,2,.... Hence, the current options for non-Hermitian problems are either 
to perform extra work (O(nk) operations at step k) and use extra storage (O(nk) words to perform 
k iterations) to find optimal approximations from the successive Krylov subspaces or to settle for 
nonoptimal approximations. The (full) GMRES (generalized minimal residual) algorithm [SS86] 
finds the approximation for which the 2-norm of the residual is minimal, at the cost of this extra 
work and storage, while other non-Hermitian iterative methods (e.g., BiCG [Fle75], CGS [Son89], 
QMR [FN91], BiCGSTAB [Vor92], and restarted GMRES [SS86], [Mor95], [DeS99]) generate 
nonoptimal approximations. 

3. Similar to the MINRES algorithm, the GMRES algorithm uses the Arnoldi iteration defined above 
to construct an orthonormal basis for the Krylov space Kx(A, ro). 

If Q; is the n by k matrix with the orthonormal basis vectors qi,... , qx as columns, then the 
Arnoldi iteration can be written simply as 


AQk = Qe Ak + Agi keAe+18k- = Qe AK. 


Here Hy is the k by k upper Hessenberg matrix with (i, j) entry equal to h;;, and H, is the k + 1 
by k matrix whose upper k by k block is Hy and whose bottom row is zero except for the last entry, 
which is Akvak- 

If qi = fo/||ro||, then the columns of Q; span the Krylov space K;(A, 19), and the GMRES 
approximation is taken to be of the form x, = Xo + Qxy, for some vector y_. To minimize the 
2-norm of the residual, the vector y, is chosen to solve the least squares problem 


a BE: — Ayyll, B= |lroll- 
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The GMRES algorithm [SS86] can be implemented as follows: 


Generalized Minimal Residual Algorithm (GMRES). 
Given x9, compute rp = b — Axo and set qi = ro/|lFoll. 


Initialize € = (1,0,... ,0)7, B = ||rol|. Fork = 1,2,..., 
Compute qui; and hj, = H(i,k),i = 1,...,k +1, using the Arnoldi algorithm. 


Apply rotations F),... , Fy; to the last column of H; that is, 
Fori =1,...,k—1, 


H(i,k) 7” Ci Sj H(i, k) 
H(i +1,k) —§; Cj HGi+1,k)] 
Compute the k™ rotation, cz and s;, to annihilate the (k + 1, k) entry of H: 


ce = |H(kK)I/ VIA A)? + ACR + 1 KP, 5k = ce H(k + 1,k)/H(k,k). 
Apply k‘" rotation to & and to last column of H: 


&(k) ce sk\ [&(k) 
eo) = & *) ( 0 


H(k,k) — cp H(k,k) +s,H(k+1,k), H(k+1,k) <0. 


If residual norm estimate 6|&(k + 1)| is sufficiently small, then 
Solve upper triangular system Hy xt Ye = B &kx1- 
Compute xx = Xp + Qkyk. 


4. The (full) GMRES algorithm described above may be impractical because of increasing storage and 
work requirements, if the number of iterations needed to solve the linear system is large. In this 
case, the restarted GMRES algorithm or one of the algorithms based on the non-Hermitian Lanczos 
process may provide a reasonable alternative. The BiCGSTAB algorithm [Vor92] is often among 
the most effective iteration methods for solving non-Hermitian linear systems. The algorithm can 
be written as follows: 


BiCGSTAB. 


Given x9, compute ro = b — Axo and set pp = fro. Choose fp such that (r9,fo) 4 0. 
Fork = 1,2,..., 


Compute Ap-1. 


(tk-1,0) 
(Apx-1fo) * 


Set Xk—1/2 = Xk-1 + @k—1Pk—1, where ay_) = 
Compute ry—1/2 = Tke-1 — 4k—1 APk-1- 


Compute Ar—1/2. 


(tk—1/2>ATk-1/2) 


Set x, = Xk—1/2 + OKT K-1/2> where wp, = aneeaa 


Compute rp = f—1/2 — Me ATk-1/2- 


Compute px = tk + bi(Px-1 — @k APk-1), Where by = “=! Hote) 


Og  (FK-15Fo) * 
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5. The non-Hermitian Lanczos algorithm can break down if (vj, wi) = 0, but neither v; nor w; is zero. 
In this case look-ahead strategies have been devised to skip steps at which the Lanczos vectors are 
undefined. See, for instance, [PTL85], [Nac91], and [FN91]. These look-ahead procedures are used 
in the QMR algorithm. 

6. When A is Hermitian and fp = ro, the BiCG method reduces to the CG algorithm, while the QMR 
method reduces to the MINRES algorithm. 

7. The question of which iterative method to use is, of course, an important one. Unfortunately, 
there is no straightforward answer. It is problem dependent and may depend also on the type of 
machine being used. If matrix-vector multiplication is very expensive (e.g., if A is dense and has 
no special properties to enable fast matrix-vector multiplication), then full GMRES is probably 
the method of choice because it requires the fewest matrix-vector multiplications to reduce the 
residual norm to a desired level. If matrix-vector multiplication is not so expensive or if storage 
becomes a problem for full GMRES, then a restarted GMRES algorithm, some variant of the QMR 
method, or some variant of BiCGSTAB may be a reasonable alternative. With a sufficiently good 
preconditioner, each of these iterative methods can be expected to find a good approximate solution 
quickly. In fact, with a sufficiently good preconditioner M, an even simpler iteration method such 
as Xk = Xx_1 + M7!(b — Axy_1) may converge in just a few iterations, and this avoids the cost of 
inner products and other things in the more sophisticated Krylov space methods. 


Applications: 


2-Norm of Residual 


10 20 30 40 50 60 70 80 90 100 

Iteration 
FIGURE 41.3 Convergence of full GMRES (solid), restarted GMRES (restarted every 10 steps) (dashed), QMR 
(dotted), and BiCGSTAB (dash-dot) for a problem from neutron transport. For GMRES (full or restarted), the number 


of matrix-vector multiplications is the same as the number of iterations, while for QMR and BiCGSTAB, the number 
of matrix-vector multiplications is twice the number of iterations. 


1. To illustrate the behavior of iterative methods for solving non-Hermitian linear systems, we have 
taken a simple problem involving the Boltzmann transport equation in one dimension: 


pe tory ob =f, xelabl wel-Lll 


where ‘ 
1 
oo = 5 f von an 
-1 
with boundary conditions 


v(bu)=W(u), -l<u<0d, 
W(a,u)=Wa(w), O< Ud. 
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The difference method used is described in [Gre97], and a test problem from [ML82] was solved. 
Figure 41.3 shows the convergence of full GMRES, restarted GMRES (restarted every 10 steps), 
QMR, and BiCGSTAB. One should keep in mind that each iteration of the QMR algorithm requires 
two matrix-vector multiplications, one with A and one with A*. Still, the QMR approximation at 
iteration k lies in the k-dimensional affine space X» + span{ro, Aro,... , A‘—!19}. Each iteration of 
the BiCGSTAB algorithm requires two matrix-vector multiplications with A, and the approximate 
solution generated at step k lies in the 2k-dimensional affine space xp + span{ro, Ato... 5 A*ly0}, 
The full GMRES algorithm finds the optimal approximation from this space at step 2k. Thus, the 
GMRES residual norm at step 2k is guaranteed to be less than or equal to the BiCGSTAB residual 
norm at step k, and each requires the same number of matrix-vector multiplications to compute. 


41.4 Preconditioners 


Definitions: 


An incomplete Cholesky decomposition is a preconditioner for a Hermitian positive definite matrix A 
of the form M = LL*, where L is a sparse lower triangular matrix. The entries of L are chosen so that 
certain entries of L L* match those of A. If L is taken to have the same sparsity pattern as the lower triangle 
of A, then its entries are chosen so that L L* matches A in the positions where A has nonzeros. 

A modified incomplete Cholesky decomposition is a preconditioner of the same form M = LL* as 
the incomplete Cholesky preconditioner, but the entries of L are modified so that instead of having M 
match as many entries of A as possible, the preconditioner M has certain other properties, such as the 
same row sums as A. 

An incomplete L U decomposition is a preconditioner for a general matrix A of the form M = LU, 
where L and U are sparse lower and upper triangular matrices, respectively. The entries of L and U are 
chosen so that certain entries of LU match the corresponding entries of A. 

A sparse approximate inverse is a sparse matrix M7! constructed to approximate A7!. 

A multigrid preconditioner is a preconditioner designed for problems arising from partial differential 
equations discretized on grids. Solving the preconditioning system Mz = r entails restricting the residual 
to coarser grids, performing relaxation steps for the linear system corresponding to the same differential 
operator on the coarser grids, and prolonging solutions back to finer grids. 

An algebraic multigrid preconditioner is a preconditioner that uses principles similar to those used for 
PDE problems on grids, when the “grid” for the problem is unknown or nonexistent and only the matrix 
is available. 


Facts: 
1. If A is an M-matrix, then for every subset S of off-diagonal indices there exists a lower triangular 
matrix L = [I;;] with unit diagonal and an upper triangular matrix U = [uj;] such that A = 
LU — R, where 


hij = 0 if (i, 7) eS, Ui; =0 if (7, 7) eS, and ‘ij =0 if (i, 7) ¢ S. 


The factors L and U are unique and the splitting A = LU — R isa regular splitting [Var60, MV77]. 
The idea of generating such approximate factorizations was considered by a number of people, one 
of the first of whom was Varga [Var60]. The idea became popular when it was used by Meijerink and 
van der Vorst to generate preconditioners for the conjugate gradient method and related iterations 
[MV77]. It has proved a successful technique in a range of applications and is now widely used 
with many variations. For example, instead of specifying the sparsity pattern of L, one might begin 
to compute the entries of the exact L-factor and set entries to 0 if they fall below some threshold 
(see, e.g., [Mun80]). 

2. For a real symmetric positive definite matrix A arising from a standard finite difference or finite 
element approximation for a second order self-adjoint elliptic partial differential equation ona grid 
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with spacing h, the condition number of A is O(h~*). When A is preconditioned using the incom- 
plete Cholesky decomposition LL’, where L has the same sparsity pattern as the lower triangle of 
A, the condition number of the preconditioned matrix L~! AL~" is still O(h-7), but the constant 
multiplying h~? is smaller. When A is preconditioned using the modified incomplete Cholesky 
decomposition, the condition number of the preconditioned matrix is O(h~!) [DKR68, Gus78]. 


. For a general matrix A, the incomplete LU decomposition can be used as a preconditioner in a 


non-Hermitian matrix iteration such as GMRES, QMR, or BiCGSTAB. At each step of the precon- 
ditioned algorithm one must solve a linear system Mz = r. This is accomplished by first solving 
the lower triangular system Ly = r and then solving the upper triangular system Uz = y. 


. One difficulty with incomplete Cholesky and incomplete LU decompositions is that the solution 


of the triangular systems may not parallelize well. In order to make better use of parallelism, sparse 
approximate inverses have been proposed as preconditioners. Here, a sparse matrix M7! is con- 
structed directly to approximate A~', and each step of the iteration method requires computation 
of a matrix-vector product z = M7'r. For an excellent recent survey of all of these preconditioning 
methods see [Ben02]. 


. Multigrid methods have the very desirable property that for many problems arising from elliptic 


PDEs the number of cycles required to reduce the error to a desired fixed level is independent of 
the grid size. This is in contrast to methods such as ICCG and MICCG (incomplete and modified 
incomplete Cholesky decomposition used as preconditioners in the CG algorithm). Early devel- 
opers of multigrid methods include Fedorenko [Fed61] and later Brandt [Bra77]. A very readable 
and up-to-date introduction to the subject can be found in [BHM00]. 

Algebraic multigrid methods represent an attempt to use principles similar to those used for PDE 
problems on grids, when the origin of the problem is not necessarily known and only the matrix 
is available. An example is the AMG code by Ruge and Stiiben [RS87]. The AMG method attempts 
to achieve mesh-independent convergence rates, just like standard multigrid methods, without 
making use of the underlying grid. A related class of preconditioners are domain decomposition 
methods. (See [QV99] and [SBG96] for recent surveys.) 


Preconditioned Algorithms 


Facts: 
1. 


It is easy to modify the algorithms of the previous sections to use left preconditioning: Simply replace 
A by M~!Aand b by M~'b wherever they appear. Since one need not actually compute M™|, this 
is equivalent to solving linear systems with coefficient matrix M for the preconditioned quantities. 
For example, letting z, denote the preconditioned residual M~!(b — Ax,), the left-preconditioned 
BiCGSTAB algorithm is as follows: 


Left-Preconditioned BiCGSTAB. 

Given xp, compute ro = b — Axo, solve Mzp = ro, and set po = Zo. 
Choose 29 such that (zo, Zp) 4 0. Fork = 1,2,..., 

Compute Apx_; and solve Mqx_1 = Apx-1- 


(Zx—1520) 
(qk—1520) * 


Compute ry_1/2 = tk-1 — @k—1 APk—1 and 2-1/2 = Zk—1 — Ak—-1Qk-1- 


Set X—1/2 = Xk-1 + @k—1Pk—1, where ay_) = 


Compute Az,_1/2 and solve Msx_1/2 = AZk—1/2- 


(Zk—1/2s8k—1/2) 
(Sk—=1/2s8k—1/2) * 


Compute rye = ty—1/2 — @¢AZ—1/2 and Ze = Zk—1/2 — OKSk-1/2: 


Set x, = Xk-1/2 + MkZk-1/2> where w, = 


= — 4k=-1 _(Zs%o) 
Compute px = Zk + bx (Pe—-1 — @rqu—-1), where by = a caese 
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2. Right or Hermitian preconditioning requires a little more thought since we want to generate 
approximations x, to the solution of the original linear system, not the modified one AM~!y = b 

or L-'AL~*y = L~'b. 


If the CG algorithm is applied directly to the problem L~! AL~*y = L~'b, then the iterates satisfy 


i (f-1 fk-1) 
Yk = Yk-1 + 4k-1Pk-1) 4-1 


(P-1, L71 AL ~*px_1) , 
f, = he = az_,L~! AL Pep 


ee i (fk, tk) 
Pk =f + Oe-1pe-1, bk-1 = => - 

(fe—-1 Pk-1) 
Defining 


= L “yy th= Lit Pe= Ll “pr, 
we obtain the following preconditioned CG algorithm for Ax = b: 


Preconditioned Conjugate Gradient Method (PCG). 


Given an initial guess x9, compute r9 = b — Ax and solve 


Mz = Yo. Set po = Zo. For k = 1,2,..., 


(For Hermitian Positive Definite Problems, with Hermitian Positive Definite Preconditioners) 
Compute Apx—1. 


Set X_ = Xk—-1 + Ap—1Pk—1, where ay_} 


— _(tk-1,2K-1) 
~~ (pk-1,Apk-1) * 
Compute rp = rh—1 — Ap APx-1. 
Solve Mz, = rx. 
Set px = Zk + bg—1Px—1, where by_) = Taeae 
Applications: 


1. Figure 41.4 shows the convergence (in terms of the 2-norm of the residual) of the ICCG algorithm 
(CG with incomplete Cholesky decomposition as a preconditioner, where the sparsity pattern of 
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FIGURE 41.4 Convergence of ICCG (dashed) and multigrid (solid) for the problem given in the introduction with 
a(x,y,z) =1+x+3yz,h = 1/50 (for ICCG), and h = 1/64 (for multigrid). 
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L was taken to match that of the lower triangle of A) and a multigrid method (using standard 
restriction and prolongation operators and the Gauss-Seidel algorithm for relaxation) for the same 
problem used in sections 2 and 3. The horizontal axis represents iterations for ICCG or cycles for 
the multigrid method. Each multigrid V-cycle costs about twice as much as an ICCG iteration, 
since it performs two triangular solves and two matrix vector multiplications on each coarse grid, 
while doing one of each on the fine grid. There is also a cost for the restriction and prolongation 
operations. The grid size was taken to be 64° for the multigrid method since this enabled easy 
formation of coarser grids by doubling h. The coarsest grid, on which the problem was solved 
directly, was taken to be of size 4 x 4 x 4. It should be noted that the number of multigrid cycles can 
be reduced by using the multigrid procedure as a preconditioner for the CG algorithm (although 
this would require a different relaxation method in order to maintain symmetry). The only added 
expense is the cost of inner products in the CG algorithm. 


41.6 Convergence Rates of CG and MINRES 


In this section, we again let A and b denote the already preconditioned matrix and right-hand side vector, 
and we assume that A is Hermitian (and also positive definite for CG). 


Facts: 


1. The CG error vector and the MINRES residual vector at step k can be written in the form 
ec = Pe (A)eo, te = Pe"(A)ro, 


where P° and P are the kth degree polynomials with value 1 at the origin that minimize the 
A-norm of the error in the CG algorithm and the 2-norm of the residual in the MINRES algorithm, 
respectively. In other words, the error e, in the CG approximation satisfies 


llexlla = min Il Pk(A)eolla 
and the residual r, in the MINRES algorithm satisfies 
Iltxl| = ni lp (A)roll, 
where the minimum is taken over all polynomials p; of degree k or less with p;(0) = 1. 
2. Let an eigendecomposition of A be written as A = UAU*, where U is a unitary matrix and 
A = diag(A,,...,A,) is a diagonal matrix of eigenvalues. If A is positive definite, define Al? to 


be UA!/?U*. Then the A-norm of a vector v is just the 2-norm of the vector A!/*v. The equalities 
in Fact 1 imply 


llexll4 = min || AY” py(A)eol| = min || pe(A)AY”eo| 


= min ||Up(A)U*A'e9|| < min Il pe(A)I|lleoll a, 


[tx] = min || Upi(A)U"ro|| = min || px(A)|| Ilroll. 


These bounds are sharp; that is, for each step k there is an initial vector for which equality holds 
[Gre79, GG94, Jou96], but the initial vector that gives equality at step k may be different from the 
one that results in equality at some other step j. 
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The problem of describing the convergence of these algorithms therefore reduces to one in 

approximation theory — How well can one approximate zero on the set of eigenvalues of A using 
a kth degree polynomial with value 1 at the origin? While there is no simple expression for the 
maximum value of the minimax polynomial on a discrete set of points, this minimax polynomial 
can be calculated if the eigenvalues of A are known and, more importantly, this sharp upper bound 
provides intuition as to what constitute “good” and “bad” eigenvalue distributions. Eigenvalues 
tightly clustered around a single point (away from the origin) are good, for instance, because the 
polynomial (1 — z/c)* is small in absolute value at all points near c. Widely spread eigenvalues, 
especially if they lie on both sides of the origin, are bad, because a low degree polynomial with value 
1 at the origin cannot be small at a large number of such points. 
Since one usually has only limited information about the eigenvalues of A, it is useful to have error 
bounds that involve only a few properties of the eigenvalues. For example, in the CG algorithm for 
Hermitian positive definite problems, knowing only the largest and smallest eigenvalues of A, one 
can obtain an error bound by considering the minimax polynomial on the interval from Xnin to 
Amax} 1.e., the Chebyshev polynomial shifted to the interval and scaled to have value 1 at the origin. 
The result is 


lela — 5 (5 ‘). | Ee ai ss = P, 
lleola | \ ve +1) © \Ve-1 Ne ley 
where K = Ainax/Amin is the ratio of largest to smallest eigenvalue of A. 

If additional information is available about the interior eigenvalues of A, one often can improve 
on this estimate while maintaining a simpler expression than the sharp bound in Fact 2. Suppose, 
for example, that A has just a few eigenvalues that are much larger than the others, say, A) <--- < 
Ane << An—ey1 < +++ < An. Consider a polynomial p, that is the product of a factor of degree £ 
that is zero at Ay—¢41,... A, and the (k — £)th degree scaled and shifted Chebyshev polynomial on 
the interval [A1,An—c]: 


2x — Ane — Ay —An-e —- Ad "Ag x 
pals) = [te ( An-e — 1 ) /e (EI II Ai ) 


i=n—l+1 


Since the second factor is zero at An—e41,..- > An and less than one in absolute value at each of the 
other eigenvalues, the maximum absolute value of this polynomial on {A1,... , An} is less than the 
maximum absolute value of the first factor on {A),... ,A,—¢}. It follows that 


ke 
pel oe 
llexlla <2 Kn—e ei a 
lleolla Vent + 1 Ay 
Analogous results hold for the 2-norm of the residual in the MINRES algorithm applied to a 


Hermitian positive definite linear system. For estimates of the convergence rate of the MINRES 
algorithm applied to indefinite linear systems see, for example, [Fis96]. 


Convergence Rate of GMRES 


Facts: 
1. 


Like MINRES for Hermitian problems, the GMRES algorithm for general linear systems produces 
a residual at step k whose 2-norm satisfies ||r,|| = minp, || px(A)roll, where the minimum is 
over all kth degree polynomials p; with p;(0) = 1. To derive a bound on this expression that is 
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independent of the direction of r9, we could proceed as in the previous section by employing an 
eigendecomposition of A. To this end, assume that A is diagonalizable and let A = VAV7 be 
an eigendecomposition, where A = diag(A,,... ,A,) is a diagonal matrix of eigenvalues and the 
columns of V are right eigenvectors of A. Then it follows that 


I[txl| = min IVp.(A)V"roll < «(V) min |l peCA)l - [Iroll, 


where «(V) = || V|| - || V~'|] is the condition number of the eigenvector matrix V. We can assume 
that the columns of V have been scaled to make this condition number as small as possible. As in 
the Hermitian case, the polynomial that minimizes || Vpx(A)V~'rol| is not necessarily the one that 
minimizes || p,(A)||, and it is not clear whether this bound is sharp. It turns out that if A isa normal 
matrix, then «(V) = 1 and the bound is sharp [GG94, Jou96]. In this case, as in the Hermitian 
case, the problem of describing the convergence of GMRES reduces to a problem in approximation 
theory — How well can one approximate zero on the set of complex eigenvalues using a kth degree 
polynomial with value 1 at the origin? 

If the matrix A is nonnormal but has a fairly well-conditioned eigenvector matrix V, then the 
above bound, while not necessarily sharp, gives a reasonable estimate of the actual size of the 
residual. In this case again, it is A’s eigenvalue distribution that essentially determines the behavior 
of GMRES. 


. In general, however, the behavior of GMRES cannot be determined from eigenvalues alone. In fact, 


it is shown in [GS94b] and [GPS96] that any nonincreasing sequence of residual norms can be 
obtained with the GMRES method applied to a problem whose coefficient matrix has any desired 
eigenvalues. Thus, for example, eigenvalues tightly clustered around 1 are not necessarily good for 
nonnormal matrices; one can construct a nonnormal matrix whose eigenvalues are equal to 1 or 
as tightly clustered around 1 as one might like, for which the GMRES algorithm makes no progress 
until step n (when it must find the exact solution). 

The convergence behavior of the GMRES algorithm for nonnormal matrices is a topic of current 
research. The analysis must involve quantities other than the eigenvalues. Some partial results have 
been obtained in terms of the field of values [EES83, Eie93, BGT05], in terms of the €-pseudospectrum 
[TE05], and in terms of the polynomial numerical hull of degree k [Gre02, Gre04]. 


Inexact Preconditioners and Finite Precision 
Arithmetic, Error Estimation and Stopping Criteria, 
Text and Reference Books 


There are a number of topics of current or recent research that will not be covered in this article. Here we 


list a few of these with sample references. 


Facts: 
1. 


The effects of finite precision arithmetic on both the convergence rate and the ultimately attainable 
accuracy of iterative methods have been studied. Example papers include [DGR95, Gre89, Gre97b, 
GRS97, GS92, GS00, Pai76, Vor90], and [Woz80]. 


2. A related topic is inexact preconditioners. Suppose the preconditioning system Mz = r is solved 


inexactly, perhaps by using an iterative method inside the outer iteration for Ax = b. Howaccurately 
should the preconditioning system be solved in order to obtain the best overall performance of 
the process? The answer is surprising. See [BF05, ES04], and [SS03] for recent discussions of this 
question. 


3. Another related idea is the use of different preconditioners at different steps of the iteration, 


sometimes called flexible iterative methods. See, for example, [Saa93]. 
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4. Finally, there is the important question of when an iterative method should be stopped. Sometimes 
one wishes to stop when the 2-norm of the error or the A-norm of the error in the CG method 
reaches a certain threshold. But one cannot compute these quantities directly. For discussions 
of estimating the A-norm of the error in the CG algorithm, as well as its connections to Gauss 
quadrature, see, for instance, [GM97, GS94, HS52], and [ST02] 

5. A number of text and reference books on iterative methods are available. These include [Axe95, 
Gre97, Hac94, Saa03], and [Vor03] 
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The eigenvalue decomposition (EVD) is an infinite iterative procedure — finding eigenvalues is equivalent 
to finding zeros of the characteristic polynomial, and, by the results of Abel and Galois, there is no algebraic 
formula for roots of the polynomial of degree greater than four. However, the number of arithmetic oper- 
ations required to compute EVD to some prescribed accuracy is also finite — EVD ofa general symmetric 
matrix requires O(n?) operations, while for matrices with special structure this number can be smaller. For 
example, the EVD ofa tridiagonal matrix can be computed in O(n?) operations (see Sections 42.5 and 42.6). 

Basic methods for the symmetric eigenvalue computations are the power method, the inverse iteration 
method, and the QR iteration method (see Section 42.1). Since direct application of those methods to a 
general symmetric matrix requires O(n*) operations, the most commonly used algorithms consist of two 
steps: the given matrix is first reduced to tridiagonal form, followed by the computation of the EVD of 
the tridiagonal matrix by QR iteration, the divide and conquer method, bisection and inverse iteration, 
or the method of multiple relatively robust representations. Two other methods are the Jacobi method, 
which does not require tridiagonalization, and the Lanczos method, which computes only a part of the 
tridiagonal matrix. 

Design of an efficient algorithm must take into account the target computer, the desired speed and 
accuracy, the specific goal (whether all or some eigenvalues and eigenvectors are desired), and the matrix size 
and structure (small or large, dense or sparse, tridiagonal, etc.). For example, if only some eigenvalues and 
eigenvectors are required, one can use the methods of Sections 42.5, 42.6, and 42.8. Ifhigh relative accuracy 
is desired and the matrix is positive definite, the Jacobi method is the method of choice. If the matrix is 
sparse, the Lanczos method should be used. We shall cover the most commonly used algorithms, like those 
which are implemented in LAPACK (see Chapter 75) and MatataB®(see Chapter 71). The algorithms 
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provided in this chapter are intended to assist the reader in understanding the methods. Since the actual 
software is very complex, the reader is advised to use professional software in practice. 

Efficient algorithms should be designed to use BLAS, and especially BLAS 3, as much as possible 
(see Chapter 74). The reasons are twofold: First, calling predefined standardized routines makes programs 
shorter and more easily readable, and second, processor vendors can optimize sets of standardized routines 
for their processor beyond the level given by compiler optimization. Examples of such optimized libraries 
are the Intel Math Kernel Library and AMD Core Math Library. Both libraries contain processor optimized 
BLAS, LAPACK, and FFT routines. 

This chapter deals only with the computation of EVD of real symmetric matrices. The need to compute 
EVD ofa complex Hermitian matrix (see Chapter 8) does not arise often in applications, and it is theoret- 
ically and numerically similar to the real symmetric case addressed here. All algorithms described in this 
chapter have their Hermitian counterparts (see e.g., [ABB99], [LSY98], and Chapters 71, 75, and 76). 

The chapter is organized as follows: In Section 42.1, we describe basic methods for EVD computations. 
These methods are necessary to understand algorithms of Sections 42.3 to 42.6. In Section 42.2, we 
describe tridiagonalization by Householder reflections and Givens rotations. In Sections 42.3 to 42.6, 
we describe methods for computing the EVD ofa tridiagonal matrix— QR iteration, the divide and conquer 
method, bisection and inverse iteration, and the method of multiple relatively robust representations, 
respectively. The Jacobi method is described in Section 42.7 and the Lanczos method is described in 
Section 42.8. For each method, we also describe the existing LAPACK or Matlab implementations. The 
respective timings of the methods are given in Section 42.9. 


42.1 Basic Methods 


Definitions: 
The eigenvalue decomposition (EVD) of a real symmetric matrix A = [aj] is given by A = UA UT, 
where U is an x n real orthonormal matrix, U'U = UU? = I,, and A = diag(Aj,...,4,) is a real 
diagonal matrix. 

The numbers A; are the eigenvalues of A, the columns u;,i = 1,...,n, of U are the eigenvectors of A, 


and Au; = A;uj,i = 1,...,n. 

If |Ai| > |A2| => +--+ = |An|, we say that 2 is the dominant eigenvalue. 

Deflation is a process of reducing the size of the matrix whose EVD is to be determined, given that one 
eigenvector is known (see Fact 4 below for details). 

The shifted matrix of the matrix A is the matrix A — pI, where wz is the shift. 

The simplest method for computing the EVD (also in the unsymmetric case) is the power method: 
given starting vector Xo, the method computes the sequences 


Vk = x, Ax, Xk41 = Ax,/|| Axx], k = 0,1,2,..., (42.1) 


until convergence. Normalization of x; can be performed in any norm and serves the numerical stability 
of the algorithm (avoiding overflow or underflow). 
Inverse iteration is the power method applied to the inverse of a shifted matrix, starting from xo: 


Ve =X, AXks Veer = (AMI) xe, Xe41 = Vegi /IIVevill k= 0,1,2,.... (42.2) 


Given starting n x p matrix Xo with orthonormal columns, the orthogonal iteration (also subspace 
iteration) forms the sequence of matrices 


Yeu1 = AXzs Yiu = Xtai Resi (QR factorization), k=0,1,2,..., (42.3) 


where X,+1 R41 is the reduced QR factorization of Yp4, (Xz41 is an n x p matrix with orthonormal 
columns and R;4+; is a upper triangular p x p matrix). 
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Starting from the matrix Ap = A, the QR iteration forms the sequence of matrices 


Ay = Qg Re (QR factorization), Ags = Re Qu, k=0,1,2,... (42.4) 


Given the shift jz, the shifted QR iteration forms the sequence of matrices 


Facts: 


Ax — WI = QgRe (QR factorization), Api, = R,Qk + ml, k=0,1,2,... (42.5) 


The Facts 1 to 14 can be found in [GV96, §8.2], [Par80, §4, 5], [SteO1, §2.1, 2.2.1, 2.2.2], and [Dem97, §4]. 


Ts 


10. 


If A, is the dominant eigenvalue and if xp is not orthogonal to wu), then in Equation 42.1 1, > Ay 
and x, — wu). In other words, the power method converges to the dominant eigenvalue and its 


eigenvector. 
| 


. The convergence of the power method is linear in the sense that 


Xx 
rial = 0 ( 


ae. 
Ay 


I ll, =O 2 
; uy; — Xl]. = — 
1 kMl2 ie 


if 


More precisely, 


where c; is the coefficient of the i-th eigenvector in the linear combination expressing the starting 
vector Xo. 


. Since A; is not readily available, the convergence is in practice determined using residuals. If 


|| Ax; — ¥¢Xz|l2 < tol, where tol is a user prescribed stopping criterion, then |A; — 1%| < tol. 


. After computing the dominant eigenpair, we can perform deflation to reduce the given EVD to the 


one of sizen —1.Let Y =[u, X] bean orthogonal matrix. Then 


A, 0 
0 A; 


> 


fa, x] A[m x] = 


where A, = X' AX. 


. The EVD of the shifted matrix A — I is given by U(A — wI)U™. Sometimes we can choose shift 


pt such that the shifted matrix A — jzI has better ratio between the dominant eigenvalue and the 
absolutely closest one, than the original matrix. In this case, applying the power method to the 
shifted matrix will speed up the convergence. 


. Inverse iteration requires solving the system of linear equations (A — wI)vy41 = xX; for vp4) in each 


step. At the beginning, we must compute the LU factorization of A — jzI, which requires 2n?/3 
operations and in each subsequent step we must solve two triangular systems, which requires 21” 
operations. 


. If wis very close to some eigenvalue of A, then the eigenvalues of the shifted matrix satisfy |A;| >> 


|Az| > --- > |An|, so the convergence of the inverse iteration method is very fast. 


. If wis very close to some eigenvalue of A, then the matrix A— jI is nearly singular, so the solutions 


of linear systems may have large errors. However, these errors are almost entirely in the direction 
of the dominant eigenvector so the inverse iteration method is both fast and accurate. 


. We can further increase the speed of convergence of inverse iterations by substituting the shift w 


with the Rayleigh quotient 1; in each step, at the cost of computing new LU factorization each time. 
See Chapter 8.2 for more information about the Rayleigh quotient. 
If 


|Ai] +++ 2 [Ap] > lApyil 2 +++ 2 [And 
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then the subspace iteration given in Equation 42.3 converges such that 
Xp > [u,... ,Up| > XP AX, > diag(A1,... Ap), 


at a speed which is proportional to |A p41/A>p ee 


11. If |Ay| > |A2| > +++ > |A,|, then the sequence of matrices A, generated by the QR iteration 
given in Equation 42.4 converges to diagonal matrix A. However, this result is not of practical 
use, since the convergence may be very slow and each iteration requires O(n) operations. Careful 
implementation, like the one described in section 42.3, is needed to construct an useful algorithm. 

12. The QR iteration is equivalent to orthogonal iteration starting with the matrix Xo = I. More 
precisely, the matrices X; from Equation 42.3 and A; from Equation 42.4 satisfy X Ps AX; = Ag. 

13. Matrices Ay and Ax, from Equations 42.4 and Equation 42.5 are orthogonally similar. In both 
cases 

Ary = Q7 Ar Qk. 

14. The QR iteration method is essentially equivalent to the power method and the shifted QR iteration 
method is essentially equivalent to the inverse power method on the shifted matrix. 

15. [Wil65, §3, 5, 6, 7] [TB97, §V] Let UAU™ and UAU' be the exact and the computed EVDs of A, 
respectively, such that the diagonals of A and A are in the same order. Numerical methods generally 
compute the EVD with the errors bounded by 

x 2 | All2 
|Ai — Ail S Gell Allo, lu; — all, < ve——__ 
mn j4; JA; = Aj\ 
where € is machine precision and ¢ and y are slowly growing polynomial functions of n which 
depend upon the algorithm used (typically O() or O(n’)). 
Examples: 
1. The eigenvalue decomposition of the matrix 
4.5013 0.6122 2.1412 2.0390 
4 0.6122 2.6210 —0.4941 —1.2164 
~ 12.1412 —0.4941 1.1543 —0.1590 
2.0390 —1.2164 —0.1590 —0.9429 
computed by the MatTLaB command [U, Lambda]=eig(A) is A = UAU? with (properly 
rounded to four decimal places) 
—0.3697 0.2496 0.1003 —0.8894 —2.3197 0 0 0 
0.2810 —0.0238 0.9593 —0.0153 7 0 0.6024 0 0 
0.3059 —0.8638 —0.1172 —0.3828}° 0 0 3.0454 0 
0.8311 0.4370 —0.2366 —0.2495 0 0 0 6.0056 
2. Let A, U, and A be as in the Example 1, and set x» = [1 1 1 1]?. The power method in 


Equation 42.1 gives x5 = [0.8893 0.0234 0.3826 0.2496]’. By setting u. = —U.4 we have 
||u, — X6||2 = 0.0081. Here (Fact 2), co = 0.7058, c; = —1.5370, and 


6 


A2 
= 0.0078. 


1 


C2 


C} 


Similarly, ||u; — X50||2 = 1.3857-107!°. However, for a different (bad) choice of the starting vector, 
X)=[0 1 0 O]7, where cy) = 0.9593 and c) = —0.0153, we have ||u, — x6||2 = 0.7956. 
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3. The deflation matrix Y and the deflated matrix A, (Fact 4) for the above example are equal to 


(correctly rounded): 
—0.8894 —0.0153 —0.3828 —0.2495 
—0.0153 0.9999 —0.0031 —0.0020 
= —0.3828 —0.0031 0.9224 —0.0506] ” 
—0.2495 —0.0020 —0.0506 0.9670 
6.0056 0 0 0 
0 2.6110 —0.6379 —1.3154 
a 0 —0.6379 0.2249 —0.8952] 
0 —1.3154 —0.8952 —1.5078 


4, Let A and xp be as in Example 2. For the shift jz = 6, the inverse iteration method in Equation 42.2 
gives ||u; — X6||2 = 6.5187 - 10~'°, so the convergence is much faster than in Example 2 (Fact 7). 
5. Let A be as in Example 1. Applying six steps of the QR iteration in Equation 42.4 gives 


6.0055 —0.0050 —0.0118 —0.0000 
—0.0050 3.0270 0.3134 0.0002 
—0.0118 0.3134 —2.3013 —0.0017} 
—0.0000 0.0002 —0.0017 0.6024 


and applying six steps of the shifted QR iteration in Equation 42.5 with p. = 6 gives 


—2.3123 0.1452 —0.0215 —0.0000 
0.1452 0.6623 0.4005 0.0000 
—0.0215 0.4005 2.9781 —0.0000] 
0.0000 0.0000 0.0000 6.0056 


6= 


In this case both methods converge. The convergence towards the matrix where the eigenvalue nearest 
to the shift can be deflated is faster for the shifted iterations. 


42.2 Tridiagonalization 


The QR iteration in Equation 42.4 in Section 42.1 and the shifted QR iteration in Equation 42.5 in Section 
42.1 require O(n3) operations (one QR factorization) for each step, which makes these algorithms highly 
unpractical. However, if the starting matrix is tridiagonal, one step of these iterations requires only O(n) 
operations. As a consequence, the practical algorithm consists of three steps: 


1. Reduce A to tridiagonal form T by orthogonal similarities, X7 AX = T. 
2. Compute the EVD of T, T = QAQ’™. 
3. Multiply U = xQ. 


The EVD of A is then A = UAU7. Reduction to tridiagonal form can be performed by using House- 
holder reflectors or Givens rotations and it is a finite process requiring O(n?) operations. Reduction to 
tridiagonal form is a considerable compression of data since an EVD of T can be computed very quickly. 
The EVD of T can be efficiently computed by various methods such as QR iteration, the divide and conquer 
method (DC), bisection and inverse iteration, or the method of multiple relatively robust representations 
(MRRR). These methods are described in subsequent sections. 
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All the following facts, except Fact 6, can be found in [Par80, §7], [TB97, pp. 196-201], [GV96, §8.3.1], 


[Ste01, 


1. 
2. 


5. 


pp. 158-162], and [Wil65, pp. 345-367]. 


Tridiagonal form is not unique (see Examples 1 and 2). 
The reduction of A to tridiagonal matrix by Householder reflections is performed as follows. Let 
us partition A as 


T 
ayy a 
A= : 
a | 
Let H be the appropriate Householder reflection (see Chapter 38.4), that is, 
vw" 
v=atsign(ay)ilabe,  H=1-2™_, 
viv 
and let . 
1 0 
H. — 
Then 
ayy | a’ H ayy | ve? oy eke 
A, AH, = ee | hi = 2 | Al? v = — sign(a))|lall2. 


This step annihilates all elements in the first column below the first subdiagonal and all elements 
in the first row to the right of the first subdiagonal. Applying this procedure recursively yields the 
triangular matrix T = X7AX, X = H,H)-++ Hy». 

H does not depend on the normalization of v. The normalization v; = 1 is useful since ap.,, can be 
overwritten by v2, and v; does not need to be stored. 

Forming H explicitly and then computing A} = HBH requires O(n’) operations, which would 
ultimately yield an O(n‘) algorithm. However, we do not need to form the matrix H explicitly — 
given v, we can overwrite B with HBH in just O(n”) operations by using one matrix-vector 
multiplication and two rank-one updates. 

The entire tridiagonalization algorithm is as follows: 


Algorithm 1: Tridiagonalization by Householder reflections 
Input: real symmetric n x n matrix A 
Output: the main diagonal and sub- and superdiagonal of A are overwritten by T, 
the Householder vectors are stored in the lower triangular part of A 
below the first subdiagonal 
for j =1:n—2 
w= sign(aj+1,j)I|Aj+in, lle 
if u ~ 0, then 
B=4jrij tu 
Vjt2:n = Aj+2:n,j/B 
endif 
4j+1j = —Kh 
Gj,j+1 = Kh 
Viti =1 
Y= —2/VigiVitin 
W = VAjaitenjtinVj4in 
q=Ww + 5VVjtin(Y]ynW) 
Aj+in,jtin = Aj+injtin + Vj4 ing? + QV} yn 
Aj+2:n,j = Vj+2:n 
endfor 
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6. [DHS89] When symmetry is exploited in performing rank-2 update, Algorithm 1 requires 4n° /3 
operations. Another important enhancement is the derivation of the block-version of the algorithm. 
Instead of performing rank-2 update on B, thus obtaining A), we can accumulate p transformations 
and perform rank-2 p update. In the first p steps, the algorithm is modified to update only columns 
and rows 1,..., p, which are needed to compute the first p Householder vectors. Then the matrix 
A is updated by A — UV" — VU", where U and V are n x p matrices. This algorithm is rich 
in matrix—matrix multiplications (roughly one half of the operations is performed using BLAS 3 
routines), but it requires extra workspace for U and V. 

7. Ifthe matrix X is needed explicitly, it can be computed from the stored Householder vectors by 
Algorithm 2. In order to minimize the operation count, the computation starts from the smallest 
matrix and the size is gradually increased, that is, the algorithm computes the sequence of matrices 


Fn-2, An-3An-2..-, X = Hy--+ Ay-2. 


Acolumn-oriented version is possible as well, and the operation count in both cases is 4n° /3. If the 
Householder matrices H; are accumulated in the order in which they are generated, the operation 
count is 2°, 


Algorithm 2: Computation of the tridiagonalizing matrix X 
Input: output from Algorithm 1 
Output: matrix X such that X7 AX = T, where A is the input of Algorithm 1 
and T is tridiagonal. 
X=I, 
for j =n—2:-1:1 
Vit = 1 
Vit2n = Aj42n,j 
Y= —2/VipieVjtin 
w= V XT pin jpunV itn 
Xj Figg hh = Xj tlin,j+lin oe Vi4 in? 
endfor 


8. The error bounds for Algorithms 1 and 2 are as follows: The matrix T computed by Algorithm 
1 is equal to the matrix, which would be obtained by exact tridiagonalization of some perturbed 
matrix A+ E (backward error), where || E ||, < yeé||Al]2 and w is a slowly increasing function of 
n. The matrix X computed by Algorithm 2 satisfies X = X + F, where ||F ll, < ¢e and @ isa 
slowly increasing function of n. 

9. Givens rotation parameters c and s are computed as in Fact 5 of Section 38.4. Tridiagonalization 
by Givens rotations is performed as follows: 


Algorithm 3: Tridiagonalization by Givens rotations 
Input: real symmetric n x n matrix A 
Output: the matrix X such that X7 AX = T is tridiagonal, main diagonal 
and sub- and superdiagonal of A are overwritten by T 
X=I, 
for j =1:n—2 
fori =j+2:n 
set x = aj41,j and y = @j,; 


compute G = : : via Fact 5 of Section 38.4 
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Algorithm 3: Tridiagonalization by Givens rotations (Continued) 


[Xan ist Xin] = [Xen jt Xin] Ge 
endfor 
endfor 


10. Algorithm 3 requires (n—1)(n—2)/2 plane rotations, which amounts to 4n? operations if symmetry 


is properly exploited. The operation count is reduced to 8n? /3 if fast rotations are used. Fast rotations 
are obtained by factoring out absolutely larger of c ands from G. 


11. The Givens rotations in Algorithm 3 can be performed in different orderings. For example, the 


elements in the first column and row can be annihilated by rotations in the planes (n — 1,n), 
(n —2,n—1),... (2,3). Since Givens rotations act more selectively than Householder reflectors, 
they can be useful if A has some special structure. For example, Givens rotations are used to 
efficiently tridiagonalize symmetric band matrices (see Example 4). 


12. Error bounds for Algorithm 3 are the same as the ones for Algorithms 1 and 2 (Fact 8), but with 


slightly different functions y and @. 


Examples: 


1; 


2: 


Algorithms 1 and 2 applied to the matrix A from Example 1 in Section 42.1 give 


4.5013 —3.0194 0 0 
T —3.0194 —0.3692 1.2804 0 
~ 0 1.2804 0.5243 —0.9303]’ 
0 0 —0.9303 2.6774 

1 0 0 0 
0 —0.2028 0.4417 —0.8740 

os 0 -—0.7091 —0.6817 —0.1800]' 
0 —0.6753 0.5833 0.4514 


Tridiagonalization is implemented in the Mattasp function T = hess(A) ([X,T] = 
hess (A) if X is to be computed, as well). In fact, the function hess is more general and it 
computes the Hessenberg form of a general square matrix. For the same matrix A as above, the 
matrices T and X computed by hess are: 


2.6562 1.3287 0 0 0.4369 0.2737 0.8569 0 

1.3287 2.4407 2.4716 0 0.7889 0.3412 —0.5112 0 

i 0 2.4716 3.1798 2.3796 |’ = —0.4322 0.8993 —0.0668 0 
0 0 2.3796 —0.9429 0 0 0 1.0000 


The block version of tridiagonal reduction is implemented in the LAPACK subroutine DSYTRD 
(file dsytrd.£). The computation of X is implemented in the subroutine DORGTR. The size of 
the required extra workspace (in elements) is !work = nb * n, where nb is the optimal block size 
(here, nb = 64), and it is determined automatically by the subroutines. The timings are given in 
Section 42.9. 
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4. Computation of Givens rotation in Algorithm 3 is implemented in the MarTLas functions 
planerot and givens, BLAS 1 subroutine DROTG, and LAPACK subroutine DLARTG. These 
implementations avoid unnecessary overflow or underflow by appropriately scaling x and y. Plane 
rotations (multiplications with G) are implemented in the BLAS 1 subroutine DROT. LAPACK 
subroutines DLAR2V, DLARGV, and DLARTV generate and apply multiple plane rotations. 
LAPACK subroutine DSBTRD tridiagonalizes a symmetric band matrix by using Givens rotations. 


42.3 Implicitly Shifted QR Method 


This method is named after the fact that, for a tridiagonal matrix, each step of the shifted QR iterations 
given by Equation 42.5 in Section 42.1 can be elegantly implemented without explicitly computing the 


shifted matrix Ay — pl. 


Definitions: 
Wilkinson’s shift j1 is the eigenvalue of the bottom right 2 x 2 submatrix of T, which is closer to ty,n. 


Facts: 
The following facts can be found in [GV96, pp. 417-422], [Ste01, pp. 163-171], [TB97, pp. 211-224], 


[Par80, §8], [Dem97, §5.3.1], and [Wil65, §8.50, 8.54]. 
T = [t;;] isa real symmetric tridiagonal matrix of order n and T= QA Q! is its EVD. 


1. The stable formula for the Wilkinson’s shift is 
ts 1 


T+ sign(t)/t? +t 


ty—1,n—-1 an tan 


2 


L= tan 


2. The following recursive function implements the implicitly shifted QR method given by Equa- 
tion 42.5: 


Algorithm 4: Implicitly shifted QR method for tridiagonal matrices 
Input: real symmetric tridiagonal n x n matrix T 
Output: the diagonal of T is overwritten by its eigenvalues 
function T = QR_iteration(T) 
repeat % one sweep 
compute a suitable shift jw 
setx = tj) —wandy = ty 


compute G = ys f via Fact 5 of Chapter 38.4 


Th 1:3 Th 1:3 
> = G > 
| ba 
Tia Tisa| = [Tian Tia] Gr? 
fori =2:n—1 


set x = tj) and y = tj41)-1 


compute G = ; via Fact 5 of Section 38.4 
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Algorithm 4: Implicitly shifted QR method for tridiagonal matrices (Continued) 


Tjj-1i+2 =G Ti,i-1i42 
Ti4ai—1i+2 Tii-1it2 


= T 
[Tass T1424] = [Tass T-1:142,41| G 
endfor 
until § [fj41| <€/ltii-taizi| forsomei % deflation 


set ti41; = Oand tj4,; = 0 
Thai = QR -iteration(T;.i,1:i) 
Tittnitin = QR iteration( T;+1:nji+1:n) 


Wilkinson’s shift (Fact 1) is the most commonly used shift. With Wilkinson’s shift, the algorithm 
always converges in the sense that t,_1,, > 0. The convergence is quadratic, that is, |[Tk+1]n—1n| < 
c|{Tk]n—1n|’ for some constant c, where 7; is the matrix after k-th sweep. Even more, the convergence 
is usually cubic. However, it can also happen that some 4; 4;,1 4 n — 1, becomes sufficiently small 
before t,—1,, so the practical program has to check for deflation at each step. 
The plane rotation parameters at the start of the sweep are computed as if the shifted matrix 
T — wl has been formed. Since the rotation is applied to the original T and not to T — pl, 
this creates new nonzero elements at the positions (3,1) and (1,3), the so-called bulge. The 
subsequent rotations simply chase the bulge out of the lower right corner of the matrix. The 
rotation in the (2,3) plane sets the elements (3,1) and (1,3) back to zero, but it generates two 
new nonzero elements at positions (4,2) and (2,4); the rotation in the (3,4) plane sets the 
elements (4,2) and (2,4) back to zero, but it generates two new nonzero elements at positions 
(5,3) and (3,5), etc. The procedure is illustrated in Figure 42.1: “x” denotes the elements that 
are transformed by the current plane rotation, “x” denotes the newly generated nonzero 
elements (the bulge), and 0 denotes the zeros that are reintroduced by the current plane 
rotation. 

The effect of this procedure is the following. At the end of the first sweep, the resulting matrix 
T, is equal to the the matrix that would have been obtained by factorizing T — ~I = QR and 
computing T; = RQ + yl as in Equation 42.5. 
Since the convergence of Algorithm 4 is quadratic (or even cubic), an eigenvalue is isolated after 
just a few steps, which requires O(n) operations. This means that O(n?) operations are needed to 
compute all eigenvalues. 
If the eigenvector matrix Q is desired, the plane rotations need to be accumulated similarly to the 
accumulation of X in Algorithm 3. This accumulation requires O(n) operations (see Example 2 
below and Fact 5 in Section 42.9). Another, usually faster, algorithm to compute Q is given in Fact 9 
in Section 42.9. 


x x * x x 0 x x x x x x 
x x x x xX xX * x x x 0 x xX x x x x 
xX xX XxX O: ie ate x x xX * x x x 0 x xX x 
x x xX * xX xX X Ox x x xX xX xX * x x x 0 
x x x x x x * xX xX x Ox x x x xX Xx 
x x x x x x * xX xX Ox x 


FIGURE 42.1 Chasing the bulge in one sweep of the implicit QR iteration for n = 6. 
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7. Thecomputed eigenvalue decomposition T = QA Q? satisfies the error bounds from Fact 15 in sec- 
tion 42.1 with A replaced by T and U replaced by Q. The deflation criterion implies |f+41| < €||T|lz, 
which is within these bounds. 

8. Combining Algorithms 1, 2, and 4 we get the the following algorithm: 


Algorithm 5: Real symmetric eigenvalue decomposition 
Input: real symmetric n x n matrix A 
Output: eigenvalue matrix A and, optionally, eigenvector matrix U of A 
if only eigenvalues are required, then 
Compute T by Algorithm 1 
T = QR-iteration(T) % Algorithm 4 


A = diag(T) 
else 
Compute T by Algorithm 1 
Compute X by Algorithm 2 
T = QR -iteration(T ) % with rotations accumulated in Q 
A = diag(T) 
U=xXQ 
endif 


9. The EVD computed by Algorithm 5 satisfies the error bounds given in Fact 15 in section 42.1. 
However, the algorithm tends to perform better on matrices, which are graded downwards, that is, 
on matrices that exhibit systematic decrease in the size of the matrix elements as we move along 
the diagonal. For such matrices the tiny eigenvalues can usually be computed with higher relative 
accuracy (although counterexamples can be easily constructed). Ifthe tiny eigenvalues are of interest, 
it should be checked whether there exists a symmetric permutation that moves larger elements to 
the upper left corner, thus converting the given matrix to the one that is graded downwards. 


Examples: 


1. For the matrix T from Example 1 in section 42.2, after one sweep of Algorithm 4, we have 


2.9561 3.9469 0 0 
3.9469 0.8069 —0.7032 0 
~ 0 —0.7032 0.5253 0.0091} 
0 0 0.0091 3.0454 


2. Algorithm 4 is implemented in the LAPACK subroutine DSTEQR. This routine can compute just 
the eigenvalues, or both eigenvalues and eigenvectors. To avoid double indices, the diagonal and 
subdiagonal entries of T are stored in one dimensional vectors, d; = T;; ande; = Tj+1,;, respectively. 
The timings are given in Section 42.9. 

3. Algorithm 5 is implemented in the Matlab routine eig. The command Lambda = eig(A) 
returns only the eigenvalues, [U, Lambda] =eig (A) returns the eigenvalues and the eigenvectors 
(see Example 1 in Section 42.1). 

4. The LAPACK implementation of Algorithm 5 is given in the subroutine DSYEV. To compute only 
eigenvalues, DSYEV calls DSYTRD and DSTEQR without eigenvector option. To compute both 
eigenvalues and eigenvectors, DSYEV calls DSYTRD, DORGTR, and DSTEQR with the eigenvector 
option. The timings are given in Section 42.9. 
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42.4 Divide and Conquer Method 


This is currently the fastest method for computing the EVD ofa real symmetric tridiagonal matrix T. It is 
based on splitting the given tridiagonal matrix into two matrices, then computing the EVDs of the smaller 
matrices and computing the final EVD from the two EVDs. The method was first introduced in [Cup81], 
but numerically stable and efficient implementation was first derived in [GE95]. 


Facts: 


The following facts can be found in [Dem97, pp. 216-228], [Ste01, pp. 171-185], and [GE95]. 
T = [tj] is a real symmetric tridiagonal matrix of order n and T= UA U" is its EVD. 


1. Let T be partitioned as 


dy ey 


€; dy e2 


er-1 Ak ek | T | 


ek ky1 kt 


€n—2 dn-1 Cn-1 


en-1 dy, 


We assume that T is unreduced, that is, e; # 0 for all i. Further, we assume that e; > 0 for all i, 
which can be easily be attained by diagonal similarity with a diagonal matrix of signs (see Example 
1 below). Let 


T=T—- CKELEL » hb=h- exe} « (42.6) 


In other words, 7; is equal to T; except that d; is replaced by dy — ex, and T is equal to T; except 
that d+, is replaced by dy41 — ex. 

OF ex 
x il 
column of uP and the first column of UF), Set U = U, @ Up and A = A, @ Ao. Then 


Let T = U;A;U;, i = 1,2, be the respective EVDs and let v = (v consists of the last 


i A =U(A+ew')U!. (42.7) 
2 


U; 


+ a 


If 


A + exvv’ = XAXT 
is the EVD of the rank-one modification of the diagonal matrix A, then T = UAU', where 
U = UX is the EVD of T. Thus, the original tridiagonal eigenvalue problem is reduced to two 
smaller tridiagonal eigenvalue problems and one eigenvalue problem for the rank-one update of a 
diagonal matrix. 
2. Ifthe matrix A + e,vv" is permuted such that hy > +++ > A,, then A; and A; are interlaced, that is, 


Me Py Say Se A ee S as 


Moreover, ifA;_ = A; for some i, then one eigenvalue is obviously known exactly, that is, A; = hi. 
In this case, A; can be deflated by applying to A +e, vv! a plane rotation in the (i — 1, i) plane, where 
the Givens rotation parameters c and s are computed from v;_; and v; as in Fact 5 of Section 38.4. 
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3. If all A; are different, then the eigenvalues i; of A + e,vv! are solutions of the so-called secular 
equation, 


n 2 
Vv: 
l+e ~_——_ = 0. 
ial Ai — A 


The eigenvalues can be computed by bisection, or by some faster zero finder of the Newton type, 
and they need to be computed as accurately as possible. 
4. Once the eigenvalues 4; of A + exvv" are known, the corresponding eigenvectors are 


xj = (A = Aj 1)'v. 


5. Each 4; and x; in Facts 3 and 4 is computed in O(n) operations, respectively, so the overall 
computational cost for computing the EVD of A + exw! is O(n’). 

6. The accuracy of the computed EVD is given by Fact 15 in section 42.1. However, if some eigenvalues 
are too close, they may not be computed with sufficient relative accuracy. As a consequence, the 
eigenvectors computed by using Fact 4 may not be sufficiently orthogonal. One remedy to this 
problem is to solve the secular equation from Fact 3 in double of the working precision. A better 
remedy is based on the solution of the following inverse eigenvalue problem. If Ay > --- > A, and 
Ay > Ay > Ad > dn > > Agit > Anni > An > Ags then A; are the exact eigenvalues of the 
matrix A + e,007, where 


a 7 TTj-1 (Aj Aj ) 
vj = signy; _ x 7 
jar jai Aj Ai) 
Instead of computing x; according to Fact 4, we compute % = (A — 4;1)7!¥. The eigenvector 
matrix of T is now computed as U = UX, where X = Ix: see x, , instead of U = UX as in Fact 


1. See also Fact 8. 
7. The algorithm for the divide and conquer method is the following: 


Algorithm 6: Divide and conquer method 

Input: real symmetric tridiagonal n x n matrix T with t;_,; > 0 for alli 
Output: eigenvalue matrix A and eigenvector matrix U of T 

function (A, U) = Divide_and_Conquer(T) 


ifn = 1, then 
U=1 
A=T 
else 


k = floor(n/2) 
form 7; and T) = as in Equation 42.6 in Fact 1 
(A,,0,) = Divide_and_Conquer(T;) 
(A, 05) = Divide_and_Conquer(T)) 
form A + exvv! as in Equation 42.7 in Fact 1 
compute the eigenvalues A; via Fact 3 
compute V via Fact 6 
X= (A a A,1)7!¥ 
Uy, a 
U= th X 
endif 


8. The rationale for the approach of Fact 6 and Algorithm 6 is the following: The computations of 
Vv and X; involve only subtractions of exact quantities, so there is no cancellation. Thus, all entries 
of each X; are computed with high relative accuracy so X; are mutually orthonormal to working 
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precision. Also, the transition from the matrix A + exvv! to the matrix A + e,007 induces only 
perturbations that are bounded by €||T||. Thus, the EVD computed by Algorithm 6 satisfies the 
error bounds given in Fact 15 in section 42.1, producing at the same time numerically orthogonal 
eigenvectors. For details see [Dem97, pp. 224-226] and [GE95]. 

Although Algorithm 6 requires O(n) operations (this is due to the computation of U in the last 
line), it is in practice usually faster than Algorithm 4 from Fact 2 in section 42.3. This is due to 
deflations which are performed when solving the secular equation from Fact 3, resulting in matrix 
X having many zeros. 


10. The operation count of Algorithm 6 can be reduced to O(n? log n) if the Fast Multipole Method, 
originally used in particle simulation, is used for solving secular equation from Fact 3 and for 
multiplying UX in the last line of Algorithm 6. For details see [Dem97, pp. 227-228] and [GE95]. 

Examples: 
1. Let T be the matrix from Example 1 in section 42.2 pre- and postmultiplied by the matrix D = 
diag(1,—1,—1, 1): 
4.5013 3.0194 0 0 
3.0194 —0.3692 1.2804 0 
at 0 1.2804 0.5243 0.9303] 
0 0 0.9303 2.6774 
The EVDs of the matrices T, and 1; from Equation 42.6 in Fact 1 are 
z 4.5013 3.0194 0 0.3784 —0.9256 A —2.8841 0 
'~ 13.0194 —1.6496] ’ ~ |-0.9256 —0.3784]° | 0 5.7358] 
Fa —0.7561 0.9303 ur —0.9693 —0.2458 nN —0.9920 0 
* | 0.9303 2.6774} ” ~ | 0.2458 -0.9693]’ ~* | 0 2.9132] 
so, in Equation 42.7 in Fact 1, we have 
A= diag(—2.8841, 5.7358, —0.9920, 2.9132), 
v=[—0.9256 —0.3784 —0.9693 —0.2458]". 
2. Algorithm 6 is implemented in the LAPACK subroutine DSTEDC. This routine can compute just the 


42.5 


eigenvalues or both, eigenvalues and eigenvectors. The routine requires workspace of approximately 
n? elements. The timings are given in Section 42.9. 


Bisection and Inverse Iteration 


The bisection method is convenient if only part of the spectrum is needed. If the eigenvectors are needed, 


as well, 


Facts: 


they can be efficiently computed by the inverse iteration method (see Facts 7 and 8 in Section 42.1). 


The following facts can be found in [Dem97, pp. 228-213] and [Par80, pp. 65-75]. 


Ais 


1. 


a real symmetric n x n matrix and T is a real symmetric tridiagonal n x n matrix. 


(Sylvester’s theorem) For a real nonsingular matrix X, the matrices A and X T AX have the same 
inertia. (See also Section 8.3.) 
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2. Leta,B € Rwith aw < £. The number of eigenvalues of A in the interval [a, 6) is equal to 
v(A — BI) — v(A — aI). By systematically choosing the intervals [a, 8), the bisection method 
pinpoints each eigenvalue of A to any desired accuracy. 

3. In the factorization T — wl = LDL", where D = diag(d),...,d,) and L is the unit lower 
bidiagonal matrix, the elements of D are computed by the recursion 


dh=ti—-yw, dj = (ti — py) bata, i=2,...m, 


and the subdiagonal elements of L are given by ]j+1; = t+41,;/d;. By Fact 1 the matrices T and D 
have the same inertia, thus the above recursion enables an efficient implementation of the bisection 
method for T. 

4. The factorization from Fact 3 is essentially Gaussian elimination without pivoting. Nevertheless, if 
d; # 0 for all i, the above recursion is very stable (see [Dem97, Lemma 5.4] for details). 

5. Even when d;_; = 0 for some /, if the IEEE arithmetic is used, the computation will continue and 
the inertia will be computed correctly. Namely, in that case, we would have dj = —oo, lj41; = 0, 
and dj41 = ti41,i41 — W. For details see [Dem97, pp. 230-231] and the references therein. 

6. Computing one eigenvalue of T by using the recursion from Fact 3 and bisection requires O(n) 
operations. For a computed eigenvalue the corresponding eigenvector is computed by inverse 
iteration given by Equation 42.2. The convergence is very fast (Fact 7 in Section 42.1), so the cost 
of computing each eigenvector is also O(m) operations. Therefore, the overall cost for computing 
all eigenvalues and eigenvectors is O(n”) operations. 

7. Both, bisection and inverse iteration are highly parallel since each eigenvalue and eigenvector can 
be computed independently. 

8. If some of the eigenvalues are too close, the corresponding eigenvectors computed by inverse 
iteration may not be sufficiently orthogonal. In this case, it is necessary to orthogonalize these 
eigenvectors (for example, by the modified Gram—Schmidt procedure). If the number of close 
eigenvalues is too large, the overall operation count can increase to O(n). 

9. The EVD computed by bisection and inverse iteration satisfies the error bounds from Fact 15 in 
Section 42.1. 


Examples: 

1. The bisection method for tridiagonal matrices is implemented in the LAPACK subroutine DSTEBZ. 
This routine can compute all eigenvalues in a given interval or the eigenvalues from A; to A,, where 
| < k, and the eigenvalues are ordered from smallest to largest. Inverse iteration (with reorthogo- 
nalization) is implemented in the LAPACK subroutine DSTEIN. The timings for computing half 
of the largest eigenvalues and the corresponding eigenvectors are given in Section 42.9. 


42.6 Multiple Relatively Robust Representations 


The computation of the tridiagonal EVD which satisfies the error bounds of Fact 15 in section 42.1 such 
that the eigenvectors are orthogonal to working precision, all in O(n”) operations, has been the “holy grail” 
of numerical linear algebra for a long time. The method of Multiple Relatively Robust Representations 
(MRRR) does the job, except in some exceptional cases. The key idea is to implement inverse iteration 
more carefully. The practical algorithm is quite elaborate and only main ideas are described here. 


Facts: 


The following facts can be found in [Dhi97], [DP04], and [DPV04]. 

T = [t;] denotes a real symmetric tridiagonal matrix of order n. D, D,, and D_ are diagonal matrices 
with the i-th diagonal entry denoted by dj, D;(i), and D_(i), respectively. L and L4 are unit lower 
bidiagonal matrices and U_ is a unit upper bidiagonal matrix, where we denote (L)j+1,; by li, (L+)i41; 
by L+(i), and (U_)j;;41 by U_(i). 
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1. Instead of working with the given T, the MRRR method works with the factorization T = L DL™ 
(computed, for example, as in Fact 3 in Section 42.5 with uw = 0). If T is positive definite, then all 
eigenvalues of L DL” are determined to high relative accuracy in the sense that small relative changes 
in the elements of L and D cause only small relative changes in the eigenvalues. If T is indefinite, 
then the tiny eigenvalues of L DL? are determined to high relative accuracy in the same sense. The 
bisection method based on Algorithms 7a and 7b computes the well determined eigenvalues of 
LDL! to high relative accuracy, that is, the computed eigenvalue i satisfies |A — A| = O(ne|Al). 

2. The MRRR method is based on the following three algorithms: 


Algorithm 7a: Differential stationary qd transform 
Input: factors L and D of T and the computed eigenvalue 4 
Output: matrices D, and L, such that LDL? — AI = L4D,L{ and vector s 


s; =X 
fori =1:n—1 
Ds(i) = 5; +d; 


L+(i) = ili)/D+(@) 
Sig. = Ly(i)lisi — 2 
endfor 


D4(n) = Sn + dy 


Algorithm 7b: Differential progressive qd transform 
Input: factors L and D of T and the computed eigenvalue A 
Output: matrices D_ and U_ such that LDL’ — AI = U_D_U! and vector p 
Pa = dn on a 
fori =n—1:-1:1 
DG+1) =dl? + pis: 
t=d;/D_(i+1) 


U_(i) =|t 

Pi = Pitt — rN 
endfor 
D_(1) = pi 


Algorithm 7c: Eigenvector computation 
Input: output of Algorithms 7a and 7b and the computed eigenvalue i 
Output: index r and the eigenvector u such that LDL™u = Au. 
fori =1:n-—1 
Vi= Sit rin Pi+1 
endfor 
Yn = Sn + Pat Xr 
find r such that |y,| = min; |y;| 


uy = 1 

fori =r—1:-1:1 
uj = —Ly(i)uini 

endfor 


fori=r:in-—1 
Ujy, = —U_(i)uj 

endfor 

u=u/|ull, 
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3. Algorithm 7a is accurate in the sense that small relative perturbations (of the order of few €) in the 
elements /;, d;, and the computed elements L,(i) and D;(i) make L DL’ -il= La Die an 
exact equality. Similarly, Algorithm 7b is accurate in the sense that small relative perturbations in 
the elements /;, d;, and the computed elements U_(i) and D_(i) make LDL? — AI =U_D_U"™ 
an exact equality. 

4. The idea behind the Algorithm 7c is the following: Index r is the index of the column of the 
matrix (L DL’ — 41)~! with the largest norm. Since the matrix L DL? — AJ is nearly singular, 
the eigenvector is computed in just one step of inverse iteration given by Equation 42.2 starting 
from the vector y,e,. Further, LDL™ — AI = NANT, where NANT is the the so-called twisted 
factorization obtained from L,, D,, U_, and D_: 


A = diag(D,(1),...,Dz(r — 1), y,, D-(r + 1),..., D_(n)), 
Nii = 1, Ni+1i = L+(2); t= 1,027 1, Nii+1 = U_(i), Y= tyacpt 1. 


Since Ae, = y,e, and Ne, = e,, solving NAN Tu = pe, is equivalent to solving N Tu =e, 
which is exactly what is done by Algorithm 7c. 

5. If an eigenvalue A is well separated from other eigenvalues in the relative sense (the quantity 
MiN,eo(A),uxa [A — I/|A| is large, say greater than 1073), then the computed vector & satisfies 
|| sin ©(u, a) ||2 = O(ne). If all eigenvalues are well separated from each other, then the computed 
EVD satisfies error bounds of Fact 15 in Section 42.1 and the computed eigenvectors are numerically 
orthogonal, that is, ja} a;| = O(ne) fori F j. 

6. If there is a cluster of poorly separated eigenvalues which is itself well separated from the rest of 
o (A), the MRRR method chooses a shift 4 which is near one end of the cluster and computes a new 
factorization LDL’ — I = L4 D,L1. The eigenvalues within the cluster are then recomputed by 
bisection as in Fact 1 and their corresponding eigenvectors are computed by Algorithms 7a, 7b, and 
7c. When properly implemented, this procedure results in the computed EVD, which satisfies the 
error bounds of Fact 15 in Section 42.1 and the computed eigenvectors are numerically orthogonal. 


Examples: 


1. The MRRR method is implemented in the LAPACK subroutine DSTEGR. This routine can compute 
just the eigenvalues, or both eigenvalues and eigenvectors. The timings are given in Section 42.9. 


42.7 Jacobi Method 


The Jacobi method is the oldest method for EVD computations [Jac846]. The method does not require 
tridiagonalization. Instead, the method computes a sequence of orthogonally similar matrices which 
converge to A. In each step a simple plane rotation which sets one off-diagonal element to zero is performed. 


Definitions: 


A isa real symmetric matrix of order x and A = UAU? is its EVD. 
The Jacobi method forms a sequence of matrices, 


Ag = A, Ag+ = G(iks jks €,5)AgG(iks jks G8)" k= Ly Qin 


where G (ix, jx, , S) is the plane rotation matrix defined in Chapter 38.4. The parameters c and s are chosen 
such that [Ag4i]i, j, = [Axsi] j,i, = 0 and are computed as described in Fact 1. 

The plane rotation with c and s as above is also called the Jacobi rotation. 

The off-norm of A is defined as off(A) = (30; © et aj)" 2, that is, off-norm is the Frobenius norm of 
the matrix consisting of all off-diagonal elements of A. 

The choice of pivot elements [ Ax];, ;, is called the pivoting strategy. 

The optimal pivoting strategy, originally used by Jacobi, chooses pivoting elements such that |[ Ax], j, | = 
maxj<; |[Ax]ijl- 
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The row cyclic pivoting strategy chooses pivot elements in the systematic row-wise order, 
(1,2), d, 3), See <F d, n), (2; 3), (2, 4), AR Par (2, n), (3,4), tes (n a 1, n). 


Similarly, the column-cyclic strategy chooses pivot elements column-wise. 
One pass through all matrix elements is called cycle or sweep. 


Facts: 
The Facts 1 to 8 can be found in [Wil65, pp. 265-282], [Par80, §9], [GV96, §8.4], and [Dem97, §5.3.5]. 
1. The Jacobi rotations parameters c and s are computed as follows: If [ Ax];,;, = 0, then c = 1 and 
s = 0, otherwise 
[Alice — [Ak] jcic A sign(T ) —_ 1 
2[ Aki; ji, , [cr] + /1+ 72° VI+ 


2. After each rotation, the off-norm decreases, that is, 


off (Anyi) = off (Ag) — [Ag]? 


ik jk 


With the appropriate pivoting strategy, the method converges in the sense that 


CoO 
off(Ay) 0, Ar A, [JR > U. 
k=1 


3. For the optimal pivoting strategy the square of the pivot element is greater than the average squared 
element, [ Ax] bik > off’(A) reat Thus, 


off?(Ansi) < (1 2 ) of? (Ax) 


2 

n(n — 1) 
and the method converges. 

4. For the row cyclic and the column cyclic pivoting strategies, the method converges. The convergence 
is ultimately quadratic in the sense that off(Ag4n(n—1)/2) < Y off?(A,) for some constant Ys 
provided off( A;) is sufficiently small. 

5. We have the following algorithm: 


Algorithm 8: Jacobi method with row-cyclic pivoting strategy 
Input: real symmetric n x n matrix A 
Output: the eigenvalue matrix A and the eigenvector matrix U 
U=I, 
repeat % one cycle 
fori =1:n—1 
for j =i+1:n 
compute c and s according to Fact 1 
Ajiin a ha 

= G(i, j,c,5) 
Ajuan Ajuan 
[Aten Auni| = [Aton Ain G(i, ah cys)? 


[Uren Urn] = [Uren Urn] Gi, } cys)! 
endfor 
endfor 
until off(A) < tol for some user defined stopping criterion tol 
A = diag(A) 
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6. Detailed implementation of the Jacobi method can be found in [Rut66] and [WR71]. 
The EVD computed by the Jacobi method satisfies the error bounds from Fact 15 in Section 42.1. 

8. The Jacobi method is suitable for parallel computation. There exist convergent parallel strategies 
which enable simultaneous execution of several rotations. 

9. [GV96, p. 429] The Jacobi method is simple, but it is slower than the methods based on tridiago- 
nalization. It is conjectured that standard implementations require O(n? log n) operations. More 
precisely, each cycle clearly requires O(n*) operations and it is conjectured that log n cycles are 
needed until convergence. 

10. [DV92], [DV05] If A is positive definite, the method can be modified such that it reaches the speed 
of the methods based on tridiagonalization and at the same time computes the eigenvalues with 
high relative accuracy. (See Chapter 46 for details.) 


N 


Examples: 
1. Let A be the matrix from Example 1 in section 42.1. After executing two cycles of Algorithm 8, we 
have 
6.0054 —0.0192 0.0031 0.0003 


—0.0192 3.0455 —0.0005 —0.0000 
0.0031 —0.0005 0.6024 —0.0000] 
0.0003 —0.0000 0.0000 —2.3197 


42.8 Lanczos Method 


If the matrix A is large and sparse and if only some eigenvalues and their eigenvectors are desired, sparse 
matrix methods are the methods of choice. For example, the power method can be useful to compute 
the eigenvalue with the largest modulus. The basic operation in the power method is matrix-vector mul- 
tiplication, and this can be performed very fast if A is sparse. Moreover, A need not be stored in the 
computer — the input for the algorithm can be just a program which, given some vector x, computes the 
product Ax. An “improved” version of the power method, which efficiently computes several eigenvalues 
(either largest in modulus or near some target value j) and the corresponding eigenvectors, is the Lanczos 
method. 


Definitions: 


A is a real symmetric matrix of order n. 
Given a nonzero vector x and an index k < n, the Krylov matrix is defined as 
K,=([x Ax Atx --- Ak-!x], 


Facts: 
The following facts can be found in [Par80, §13], [GV96, §9], [Dem97, $7], and [Ste01, §5.3]. 


1. The Lanczos method is based on the following observation. If Ky = XR is the QR factorization 
of the matrix K; (see Sections 5.5 and 38.4), then the k x k matrix T = X' AX is tridiagonal. 
The matrices X and T can be computed by using only matrix-vector products in just O(kn) 
operations. Let T = QA Q! be the EVD of T (computed by any of the methods from Sections 42.3 
to 42.6). Then A; approximate well some of the largest and smallest eigenvalues of A. The columns 
of the matrix U = XQ approximate the corresponding eigenvectors of A. We have the following 
algorithm: 


42-20 


25 


4. 


Handbook of Linear Algebra 


Algorithm 9: Lanczos method 
Input: real symmetric n x n matrix A, unit vector x and index k < n 
Output: matrices A and U 
X.) =X 
fori=1:k 
Z=A X:j 
a Xi Z 
ifi = 1, then 
Z=2Z-— tj X:j 
else 
Z=2Z— tj Xj — tii Xsi-1 
endif 
= |lzll2 
if u = 0, then 
stop 
else 
G+1i = Mh 
Git = bh 
Xyi41 = 2Z/p 
endif 
endfor 
compute the EVD of the tridiagonal matrix, T(1:k,1:k) = QA Qr 
U=XQ 


As j increases, the largest (smallest) eigenvalues of the matrix T,,;,1.; converge towards some of the 
largest (smallest) eigenvalues of A (due to the Cauchy interlace property). The algorithm can be 
redesigned to compute only largest or smallest eigenvalues. Also, by using shift and invert strategy, 
the method can be used to compute eigenvalues near some specified value. In order to obtain better 
approximations, k should be greater than the number of required eigenvalues. On the other side, 
in order to obtain better accuracy and efficacy, k should be as small as possible (see Facts 3 and 
4 below). 


. The eigenvalues of A are approximated from the matrix T;.,,1.4, thus, the last element v = t41,4 


is not needed. However, this element provides key information about accuracy at no extra com- 
putational cost. The exact values of residuals are as follows: || AU — UA||, = v and, in particular, 
|| AU.; — Ai U- illo = viqul,i = 1,...,k. Further, there are k eigenvalues Aqs.-+ > Ax Of A such that 
|A; — A;| < v. For the corresponding eigenvectors, we have sin 2O(u;, ;) < 2v/ minjz; |A; — dj |. 
In practical implementations of Algorithm 9, v is usually used to determine the index k. 
Although theoretically very elegant, the Lanczos method has inherent numerical instability in the 
floating point arithmetic, and so it must be implemented carefully (see, e.g., [LSY98]). Since the 
Krylov vectors are, in fact, generated by the power method, they converge towards an eigenvector 
of A. Thus, as k increases, the Krylov vectors become more and more parallel. As a consequence, 
the recursion in Algorithm 9, which computes the orthogonal bases X for the subspace range K;, 
becomes numerically unstable and the computed columns of X cease to be sufficiently orthogonal. 
This affects both the convergence and the accuracy of the algorithm. For example, it can happen 
that T has several eigenvalues which converge towards some simple eigenvalue of A (these are the 
so called ghost eigenvalues). 

The loss of orthogonality is dealt with by using the full reorthogonalization procedure. In each 
step, the new z is orthogonalized against all previous columns of X. In Algorithm 9, the formula 
Z= 2— tj; X.; — tig) X.j-1 is replaced by z = z — we XG) XG 7): To obtain better 
orthogonality, the latter formula is usually executed twice. 
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The full reorthogonalization raises the operation count to O(k?n). The selective reorthogonal- 

ization is the procedure in which the current z is orthogonalized against some selected columns of 
X. This is the way to attain sufficient numerical stability and not increase the operation count too 
much. The details of selective reorthogonalization procedures are very subtle and can be found in 
the references. (See also Chapter 44.) 
The Lanczos method is usually used for sparse matrices. Sparse matrix A is stored in the sparse 
format in which only values and indices of nonzero elements are stored. The number of operations 
required to multiply some vector by A is also proportional to the number of nonzero elements. 
(See also Chapter 43.) 


Examples: 
1. Let A be the matrix from Example 1 in section 42.1 and let x = [1/2 1/2 1/2 1/2]". For 
k = 2, the output of Algorithm 9 is 
—0.4032 —0.8804 
—2.0062 0.4842 —0.2749 
7 5.7626| ° | 0.3563 —0.3622 
0.6899 —0.1345 
with v = 1.4965 (c.f. Fact 3). For k = 3, the output is 
0.3829 —0.0244 0.8982 
—2.3107 0 0 
—0.2739 —0.9274 0.0312 
A= 0 2.8641 0 P = > 
—0.3535 —0.1176 0.3524 
0 0 5.9988 
—0.8084 0.3541 0.2607 
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with v = 0.6878. 

The Lanczos method is implemented in the ARPACK routine DSDRV*, where * denotes the com- 
putation mode [LSY98, App. A]. The routines from ARPACK are implemented in the Matias 
command eigs. Generation of a sparse symmetric 10,000 x 10,000 matrix with 10% nonzero 
elements with the MATLAB command A=sprandsym(10000,0.1) takes 15 seconds on a pro- 
cessor described in Fact 1 in secton 42.9. The computation of 100 largest eigenvalues and the cor- 
responding eigenvectors with [U, Lambda] =eigs(A,100,'LM',opts) takes 140 seconds. 
Here, index k = 200 is automatically chosen by the algorithm. (See also Chapter 76.) 


Comparison of Methods 


In this 


section, we give timings for the LAPACK implementations of the methods described in Sections 


42.2 to 42.6. The timing for the Lanczos method is given in Example 2 in Section 42.8. 


Definitions: 


A measure of processor’s efficacy or speed is the number of floating-point operations per second (flops). 


Facts: 


Ais ann x nreal symmetric matrix and A = UAU? is its EVD. T isa tridiagonal n x n real symmetric 
matrix and T = QAQ? is its EVD. T = X" AX is the reduction of A toa tridiagonal from Section 42.2. 


dl. 


Our tests were performed on the Intel Xeon processor running at 2.8 MHz with 2 Mbytes of cache 
memory. This processor performs up to 5 Gflops (5 billion operations per second). The peak 
performance is attained for the matrix multiplication with the BLAS 3 subroutine DGEMM. 
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TABLE 42.1 Execution times(s) for LAPACK routines for various matrix dimensions n. 


Routine Input Output Example n=500 n=1000 n= 2000 
DSYTRD A T 2.3 0.10 0.78 ie) 
DSYTRD/DORGTR T,X 0.17 1.09 8.6 
DSTEQR T A 3.2 0.03 0.11 0.44 
A,Q 0.32 2.23 15.41 
DSYEV A A 3.4 0.12 0.85 5.63 
A,U 0.46 3.13 22.30 
DSTEDC T A 4.2 0.02 0.08 0.28 
A, Q 0.05 0.12 0.36 
DSTEBZ T A 5.1 0.21 0.81 3.15 
DSTEIN Q 0.04 0.17 0.72 
DSTEGR £ A 6.1 0.07 0.25 0.87 
A, Q 0.09 0.35 1.29 


2. Our test programs were compiled with the Intel ifort FORTRAN compiler (version 9.0) and 
linked with the Intel Math Kernel Library (version 8.0.2). 

3. Timings for the methods are given in Table 42.1. The execution times for DSTEBZ (bisection) and 
DSTEIN (inverse iteration) are for computing one half of the eigenvalues (the largest ones) and the 
corresponding eigenvectors, respectively. 

4, The performance attained for practical algorithms is lower than the peak performance from Fact 
1. For example, by combining Facts 6 and 7 in Section 42.2 with Table 42.1, we see that the 
tridiagonalization routines DSYTRD and DORGTR attain the speed of 2 Gflops. 

5. The computation times for the implicitly shifted QR routine, DSTEQR, grow with n? when only 
eigenvalues are computed, and with n> when eigenvalues and eigenvectors are computed, as pre- 
dicted in Facts 5 and 6 in Section 42.3. 

6. The execution times for DSYEV are approximately equal to the sums of the timings for DSYTRD 
(tridiagonalization), DORGTR (computing X), and DSTEQR with the eigenvector option (com- 
puting the EVD of T). 

7. The divide and conquer method, implemented in DSTEDC, is the fastest method for computing 
the EVD ofa tridiagonal matrix. 

8. DSTEBZ and DSTEIN (bisection and inverse iteration) are faster, especially for larger dimensions, 
than DSTEQR (tridiagonal QR iteration), but slower than DSTEDC (divide and conquer) and 
DSTEGR (multiple relatively robust representations). 

9. Another algorithm to compute the EVD of T is to use DSTEQR to compute only the eigenvalues 
and then use DSTEIN (inverse iteration) to compute the eigenvectors. This is usually considerably 
faster than computing both, eigenvalues and eigenvectors, by DSTEQR. 

10. The executions times for DSTEGR are truly proportional to O(n”). 
11. The new LAPACK release, in which some of the above mentioned routines are improved with 
respect to speed and/or accuracy, is announced for the second half of 2006. 
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The definitions and basic properties of eigenvalues and eigenvectors are given in Section 4.3. A natural 
generalization is presented here in Section 43.1. Algorithms for computation of eigenvalues, eigenvectors, 
and their generalizations will be discussed in Sections 43.2 and 43.3. Although the characteristic equation 
is important in theory, it plays no role in practical eigenvalue computations. 

If a large fraction of a matrix’s entries are zeros, the matrix is called sparse. A matrix that is not sparse 
is called dense. Dense matrix techniques are methods that store the matrix in the conventional way, as 
an array, and operate on the array elements. Any matrix that is not too big to fit into a computer’s main 
memory can be handled by dense matrix techniques, regardless of whether the matrix is dense or not. 
However, since the time to compute the eigenvalues of an n x n matrix by dense matrix techniques is 
proportional to n°, the user may have to wait awhile for the results if is very large. Dense matrix techniques 
do not exploit the zeros in a matrix and tend to destroy them. With modern computers, dense matrix 
techniques can be applied to matrices of dimension up to 1000 or more. Ifa matrix is very large and sparse, 
and only a portion of the spectrum is needed, sparse matrix techniques (Section 43.3) are preferred. 

The usual approach is to preprocess the matrix into Hessenberg form and then to effect a similarity 
transformation to triangular form: T = S7! AS by an iterative method. This yields the eigenvalues of 
A as the main-diagonal entries of T. For k = 1,...,  — 1, the first k columns of S span an invariant 
subspace. The eigenvectors of an upper-triangular matrix are easily computed by back substitution, and 
the eigenvectors of A can be deduced from the eigenvectors of T [GV96, § 7.6], [Wat02, § 5.8]. Ifa matrix 
Ais very large and sparse, only a partial similarity transformation is possible because a complete similarity 
transformation would require too much memory and take too long to compute. 


43.1 The Generalized Eigenvalue Problem 


Many matrix eigenvalue problems are most naturally viewed as generalized eigenvalue problems. 


Definitions: 
Given A € C”*" and B € C”*", the nonzero vector v € C" is called an eigenvector of the pair (A, B) if 
there are scalars jz, v € C, not both zero, such that 


vAv = wBv. 
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Then, the scalar A = jx/v is called the eigenvalue of (A, B) associated with the eigenvector v. If v = 0, 
then the eigenvalue is oo by convention. 

The expression A — x B, with indeterminate x, is called a matrix pencil. Whether we refer to the pencil 
A — xB or the pair (A, B), we are speaking of the same object. The pencil (or the pair (A, B)) is called 
singular if A — 1B is singular for all A € C. The pencil is regular if there exists a A € C such that A—AB 
is nonsingular. We will restrict our attention to regular pencils. 

The characteristic polynomial of the pencil A — xB is det(xB — A), and the characteristic equation 
is det(xB — A) = 0. 

Two pairs (A, B) and (C, D) are strictly equivalent if there exist nonsingular matrices S; and Sj such 
that C — AD = S\(A — AB)S) for all A € C. If S; and S) can be taken to be unitary, then the pairs are 
strictly unitarily equivalent. 

A pair (A, B) is called upper triangular if both A and B are upper triangular. 


Facts: 
The following facts are discussed in [GV96, § 7.7] and [Wat02, § 6.7]. 


1. When B = I, the generalized eigenvalue problem for the pair (A, B) reduces to the standard 
eigenvalue problem for the matrix A. 
. A is an eigenvalue of (A, B) if and only if A — AB is singular. 
A is an eigenvalue of (A, B) if and only if ker(AB — A) 4 {0}. 
. The eigenvalues of (A, B) are exactly the solutions of the characteristic equation det(x B — A) = 0. 
. The characteristic polynomial det(x B — A) is a polynomial in x of degree < n. 
. The pair (A, B) (or the pencil A — xB) is singular if and only if det(AB — A) = 0 for all 4, if and 
only if the characteristic polynomial det(x B — A) is equal to zero. 
7. Ifthe pair (A, B) is regular, then det(x B — A) is anonzero polynomial of degree k < n. (A, B) has 
k finite eigenvalues. 
8. The degree of det(xB — A) is exactly n if and only if B is nonsingular. 
9. If B is nonsingular, then the eigenvalues of (A, B) are exactly the eigenvalues of the matrices AB™! 
and B-1A. 
10. IfA 4 0, then A is an eigenvalue of (A, B) if and only if A~! is an eigenvalue of (B, A). 
11. Zero is an eigenvalue of (A, B) if and only if A is a singular matrix. 
12. Infinity is an eigenvalue of (A, B) if and only if B is a singular matrix. 
13. Two pairs that are strictly equivalent have the same eigenvalues. 
14. If C-—AD = S\(A—AB)Sb, then vis an eigenvector of (A, B) ifand only if Sy!v is an eigenvector 
of (C, D). 
15. (Schur’s Theorem) Every A € C"*" is unitarily similar to an upper triangular matrix S. 


16. (Generalized Schur Theorem) Every pair (A, B) is strictly unitarily equivalent to an upper triangular 


pair (S, T). 
17. The characteristic polynomial of an upper triangular pair (S, T) is [ [Qt — sxx). The eigenvalues 
k=l 
of (S, T) are A, = sxe / te, k = 1,..., 0. If he = Oand sx, 4 0, then A, = oo. If th, = 0 and 
Sx = 0 for some k, the pair (S, T) is singular. 
Examples: 
1 2 1 2 i : : : 
1. Let A = 3 4 and B = 0 411° Then the characteristic polynomial of the pair (A, B) is 


x? +x —2=0,and the eigenvalues are 1 and —2. 
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2. Since the pencil 


is upper triangular, its characteristic polynomial is (5x — 2)(3x — 7), and its eigenvalues are 2/5 


and 7/3. 
0 0 1 0 
rime oie he 


3. The pencil 
has characteristic equation x = 0. It is a regular pencil with eigenvalues 0 and oo. 


43.2 Dense Matrix Techniques 


The steps that are usually followed for solving the unsymmetric eigenvalue problem are preprocessing, 
eigenvalue computation with the QR Algorithm, and eigenvector computation. 

The most widely used public domain software for this problem is from LAPACK [ABB99], and Chapter 
75. Versions in FORTRAN and C are available. The most popular proprietary software is MATLAB, which 
uses computational routines from LAPACK. Several of LAPACK’s computational routines will be men- 
tioned in this section. LAPACK also has a number of driver routines that call the computational routines 
to perform the most common tasks, thereby making the user’s job easier. A very easy way to use LAPACK 
routines is to use MATLAB. 

This section presents algorithms for the reader’s edification. However, the reader is strongly advised to 
use well-tested software written by experts whenever possible, rather than writing his or her own code. 
The actual software is very complex and addresses details that cannot be discussed here. 


Definitions: 


A matrix A € C”*" is called upper Hessenberg if a;; = 0 whenever i > j + 1. This means that every 
entry below the first subdiagonal of A is zero. An upper Hessenberg matrix is called unreduced upper 
Hessenberg if aj.1,; 4 0 forj =1,...,n—1. 


Facts: 
The following facts are proved in [Dem97], [GV96], [Kre05], or [Wat02]. 


1. Preprocessing is a two step process involving balancing the matrix and transforming by unitary 
similarity to upper Hessenberg form. 

2. The first step, which is optional, is to balance the matrix. The balancing operation begins by 
performing a permutation similarity transformation that exposes any obvious eigenvalues. The 
remaining submatrix is irreducible. It then performs a diagonal similarity transformation D~' AD 
that attempts to make the norms of the ith row and ith column as nearly equal as possible, i = 1, 
..., 1. This has the effect of reducing the overall norm of the matrix and in diminishing the effects 
of roundoff errors [Osb60]. The scaling factors in D are taken to be powers of the base of floating 
point arithmetic (usually 2). No roundoff errors are caused by this transformation. 

3. All modern balancing routines, including the code GEBAL in LAPACK, are derived from the code 
in Parlett and Reinsch [PR69]. See also [Kre05]. 
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Algorithm 1: Balancing an Irreducible Matrix. An irreducible matrix A € C”*" is input. On 
output, A has been overwritten by D~! AD, where D is diagonal. 


b + base of floating point arithmetic (usually 2) 
D<I, 
done < 0 


while done = 0 
done < 1 


for j=1:n 
c<— Vig; lab 1 <— Veg; lajel 
s<ctr f<l 
while bc <r 
le < be, r<r/b, f<bf 
while ,br <c 
[e <c/b, r<br, f <—f/b 
ife +r <0.95s 
done <0, djj <— f djj 
Ainj <— f Ainjs Ajan <— G/f) Ajin 


end 


4. In most cases balancing will have little effect on the outcome of the computation, but sometimes 
it results in greatly improved accuracy [BDD00, § 7.2]. 

5. The second preprocessing step is to transform the matrix to upper Hessenberg form. This is ac- 
complished by a sequence of n — 2 steps. On the jth step, zeros are introduced into the jth column. 

6. For every x € C" there is a unitary matrix U such that Ux = we), for some scalar a € C, where e, 
is the vector having a 1 in the first position and zeros elsewhere. U can be chosen to be a rank-one 
modification of the identity matrix: U = I + uv’. (See Section 38.4 for a discussion of Householder 
and Givens matrices.) 


Algorithm 2. Unitary Similarity Transformation to Upper Hessenberg Form. A general matrix 
A € C”*" is input. On output, A has been overwritten by an upper Hessenberg matrix Q* AQ. 
The unitary transforming matrix Q has also been generated. 


Q< I, 
forj =1l:n—2 
Let x = Ajsin,j € cr. 
Build unitary U € C"-/*"~/ such that U*x = yey. 
A j-4i:n,j:n <_ U* A jatingjsn 
Ain, jtiin — Atn,j+inU 


Qin, jtlin <—_— Qi:n,j+1:nU 
end 


8. The cost of the reduction to Hessenberg form is proportional to n> for large n; that is, it is O(n). 

9. For large matrices, efficient cache use can be achieved by processing several columns at a time. 
This allows the processor(s) to run at much closer to maximum speed. See [GV96, p. 225], 
[Wat02, p. 210], and the LAPACK code GEHRD [ABB99]. 
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10. 


11. 


12. 


13. 


14. 


15. 
16. 


17. 


18. 


19, 


20. 


21. 


22. 


Once the matrix is in upper Hessenberg form, if any of the subdiagonal entries aj+1,; is zero, 
the matrix is block upper triangular with a j x j block and an n — j x n — j block, and the 
eigenvalue problem decouples to two independent problems of smaller size. Thus, we always work 
with unreduced upper Hessenberg matrices. 

In practice we set an entry aj+1,; to zero whenever 


lajaijl < €(ajjl + lajoij+il), 


where € is the computer’s unit roundoff. 

If T ¢ C”*" is upper triangular and nonsingular, then T~' is upper triangular. If H € C"*” is 
upper Hessenberg, then TH, HT, and THT™! are upper Hessenberg. 

The standard method for computing the eigenvalues of a Hessenberg matrix is the QR algorithm, 
an iterative method that produces a sequence of unitarily similar matrices that converges to upper 
triangular form. 

The most basic version of the QR algorithm starts with Ay = A, an unreduced upper Hessenberg 
matrix, and generates a sequence (A,,) as follows: Given A,,-1, a decomposition Am—1 = QmRm, 
where Q,, is unitary and R,,, is upper triangular, is computed. Then the factors are multiplied back 
together in reverse order to yield Ay, = Rm Qm. Equivalently Ay, = Q*, Am—1Qm. 

Upper Hessenberg form is preserved by iterations of the QR algorithm. 

The QR algorithm can also be applied to non-Hessenberg matrices, but the operations are much 
more economical in the Hessenberg case. 

The basic QR algorithm converges slowly, so shifts of origin are used to accelerate convergence: 


Am-1 = Mnl = QinRns RnQm + Mn =, Q* Am—1 Qin = Am) 


where [4m € C is a shift chosen to approximate an eigenvalue. 
Often it is convenient to take several steps at once: 


Algorithm 3. Explicit QR iteration of degree k. 
Choose k shifts 11, ... [g. 

Let p(A) = (A— 1 I)(A — pal)» (A— pl). 
Compute a QR decomposition p(A) = QR. 
A< Q*AQ 


A QR iteration of degree k is equivalent to k iterations of degree 1 with shifts j1),... , 44 applied in 
succession in any order [Wat02]. Upper Hessenberg form is preserved. In practice k is never taken 
very big; typical values are 1, 2, 4, and 6. 

One important application of multiple steps is to complex shifts applied to real matrices. Complex 
arithmetic is avoided by taking k = 2 and shifts related by 2 = 71,. 

The usual choice of k shifts is the set of eigenvalues of the lower right-hand k x k submatrix 
of the current iterate. With this choice of shifts at each iteration, the entry a,—441n-« typically 
converges to zero quadratically [WE91], isolating a k x k submatrix after only a few iterations. 
However, convergence is not guaranteed, and failures do occasionally occur. No shifting strategy that 
guarantees convergence in all cases is known. For discussions of shifting strategies and convergence 
see [Wat02] or [WE91]. 

After each iteration, all of the subdiagonal entries should be checked to see if any of them can be 
set to zero. The objective is to break the big problem into many small problems in as few iterations 
as possible. Once a submatrix of size 1 x 1 has been isolated, an eigenvalue has been found. The 
eigenvalues of a 2 x 2 submatrix can be found by careful use of the quadratic formula. Complex 
conjugate eigenvalues of real matrices are extracted in pairs. 
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23. 


24, 


253 


26. 


27. 


28. 


29. 
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The explicit QR iteration shown above is expensive and never used in practice. Instead the iteration 
is performed implicitly. 


Algorithm 4: Implicit QR iteration of degree k (chasing the bulge). 
Choose k shifts (41, .. . dk. 
x<e, % first column of identity matrix 
forj =1:k 
x < (A— py!)x 
end %xis the first column ofp(A). 
KOK MXkp2n = 0 
Letthe Cr Be unitary with U*x = we, 
Arktiin — U* Aretiin 
Ainkt! — Atnik¢1U 
Return A to upper Hessenberg form as in Algorithm 2 (Fact 7). 


The initial transformation in the implicit QR iteration disturbs the upper Hessenberg form of A, 
making a bulge in the upper left-hand corner. The size of the bulge is equal to k. In the case k = 2, 
the pattern of nonzeros is 


* OR OK OK OK OOK 
* OR OK Kk OK OK 
* OR OK OK OK OO 
* OR OK OK OK 
* OK Ok 

* OK 


The subsequent reduction to Hessenberg form chases the bulge down through the matrix and off 
the bottom. The equivalence of the explicit and implicit QR iterations is demonstrated in [GV96, 
§ 7.5] and [Wat02, § 5.7]. For this result it is crucial that the matrix is unreduced upper Hessenberg. 
For a fixed small value of k, the implicit QR iteration requires only O(n?) work. Typically only a 
small number of iterations, independent of n, are needed per eigenvalue found; the total number 
of iterations is O(n). Thus, the implicit QR algorithm is considered to be an O(n?) process. 

The main unsymmetric QR routine in LAPACK [ABB99] is HSEQR, a multishift implicit QR 
algorithm with k = 6. For processing small submatrices (50 x 50 and under), HSEQR calls 
LAHQR, a multishift QR code with k = 2. Future versions of LAPACK will include improved QR 
routines that save work by doing aggressive early deflation [BBM02b] and make better use of cache 
by chasing bulges in bunches and aggregating the transforming matrices [BBM02a]. 

If eigenvectors are wanted, the aggregate similarity transformation matrix S, the product of all 
transformations from start to finish, must be accumulated. T = S~! AS, where A is the original 
matrix and T is the final upper triangular matrix. In the real case, T will not quite be upper 
triangular. It is quasi-triangular with a 2 x 2 block along the main diagonal for each complex 
conjugate pair of eigenvalues. This causes complications in the descriptions of the algorithms, but 
does not cause any practical problems 

The eigenvectors of T are computed by back substitution [Wat02, § 5.8]. For each eigenvector x of 
T, Sx is an eigenvector of A. The total additional cost of the eigenvector computation is O(n). In 
LAPACK these tasks are performed by the routines HSEQR and TREVC. 

Invariant subspaces can also be computed. The eigenvalues of A are 4) = fi1,.-. 5 An = thn. IfA1, 
... A, are disjoint from Ax41,... 5 An, then, because T is upper triangular, the first k columns of 
S span the invariant subspace associated with {A,,... , Ax}. 
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30. 


31. 


32. 


33. 


34, 


35. 


36. 


Ifan invariant subspace associated with k eigenvalues that are not at the top of T is wanted, then those 
k eigenvalues must be moved to the top by a sequence of swapping operations. Each operation is a 
unitary similarity transformation that reverses the positions of two adjacent main-diagonal entries 
of T. The transformations are applied to S as well. Once the desired eigenvalues have been moved 
to the top, the first k columns of the transformed S span the desired invariant subspace. For details 
see [BD93] and [GV96, § 7.6]. In LAPACK these tasks are performed by the routines TREXC and 
TRSEN. 

An important difference between the symmetric and unsymmetric eigenvalue problems is that in 
the unsymmetric case, the eigenvalues can be ill conditioned. That is, a small perturbation in the 
entries of A can cause a large change in the eigenvalues. Suppose A is an eigenvalue of A of algebraic 
multiplicity 1, and let E be a perturbation that is small in the sense that ||E ||, < || All2. Then 
A+ E has an eigenvalue 4 + 6 near A. A condition number for A is the smallest number « such 
that 


15] < K||E|l2 


for all small perturbations E. Ifx and y are eigenvectors of A and A’, respectively, associated with 
A, then [Wat02, § 6.5] 


Ixll2 Ilyll 
ly? x| 


DN 
ny 


If« >> 1, A is ill conditioned. If « is not much bigger than 1, A is well conditioned. 

Condition numbers can also be defined for eigenvectors and invariant subspaces [GV96, § 7.2], 
[Wat02, § 6.5]. Eigenvectors associated with a tight cluster of eigenvalues are always ill conditioned. A 
more meaningful object is the invariant subspace associated with all of the eigenvalues in the cluster. 
This space will usually be well conditioned, even though the eigenvectors are ill conditioned. The 
LAPACK routines TRSNA and TRSEN compute condition numbers for eigenvalues, eigenvectors, 
and invariant subspaces. 


The invariant subspace associated with {A,,... ,A,} will certainly be ill conditioned if any of the 
eigenvalues Ar41,... 5 An are close to any of Aj,... , Ax. A necessary (but not sufficient) condition 
for well conditioning is that A), ... , Ax be well separated from Ax41,... , An. A related practical 


fact is that if two eigenvalues are very close together, it may not be possible to swap them stably by 
LAPACK’s TREXC. 

(Performance) A 3.0 GHz Pentium 4 machine with 1 GB main memory and 1 MB cache computed 
the complete eigensystem of a random 1000 x 1000 real matrix using MATLAB in 56 seconds. This 
included balancing, reduction to upper Hessenberg form, triangularization by the QR algorithm, 
and back solving for the eigenvectors. All computed eigenpairs (A,v) satisfied || Av — Av||) < 
10~ || Allillvlhi. 

(Generalized eigenvalue problem) The steps for solving the dense, unsymmetric, generalized eigen- 
value problem Av = A Bvare analogous to those for solving the standard problem. First (optionally) 
the pair (A, B) is balanced (by routine GGBAL in LAPACK). Then it is transformed by a strictly 
unitary equivalence to a condensed form in which A is upper Hessenberg and B is upper trian- 
gular. Then the QZ algorithm completes the reduction to triangular form. Details are given in 
[GV96, § 7.7] and [Wat02, § 6.7]. In LAPACK, the codes GGHRD and HGEQZ reduce the pair to 
Hessenberg-triangular form and perform the QZ iterations, respectively. 

Once A has been reduced to triangular form, the eigenvalues are 1; = ajj/bjj,j = 1....,n. 
The eigenvectors can be obtained by routines analogous to those used for the standard problem 
(LAPACK codes TGEVC and GGBAK), and condition numbers can be computed (LAPACK codes 
TGSNA and TGSEN). 
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Examples: 


1. The matrix 


—5.5849 x 10-°! 
—7.1724 x 10~° 
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—2.4075 x 10+” —6.1644x 10+! 6.6275 x 10+ 
—2.1248 x 107° —3.6183 x 10+ 2.6435 x 10~% 


A = 
—4.1508 x 107'© —2.1647 x 107°” 1.6229 x 107°! —7.6315 x 107!4 


4.3648 x 10-8 1.2614 x 10+°° §=—1.1986 x 10+! ~=—6.2002 x 107°! 


was balanced by Algorithm 1 (Fact 3) to produce 


—0.5585 —0.3587 —1.0950 0.1036 

—0.4813, —2.1248 —0.4313 2.7719 

—0.2337 —1.8158 0.1623 —0.6713|° 
0.2793 1.2029 —1.3627 —0.6200 


B= 


. The matrix B of Example 1 was reduced to upper Hessenberg form by Algorithm 2 (Fact 7) to yield 


—0.5585 0.7579 0.0908 —0.8694 

0.6036 —3.2560 —0.0825 —1.8020 
0 0.9777 1.2826 —0.8298]° 

0 0 —1.5266 —0.6091 


n= 


. Algorithm 4 (Fact 23) was applied to the matrix H of Example 2, with k = 1 and shift w) = hag = 
—0.6091, to produce 


—3.1238 —0.5257 1.0335 1.6798 
—1.3769 0.3051 —1.5283 0.1296 
0 —1.4041 0.3261 —1.0462]° 
0 0 —0.0473 —0.6484 


The process was repeated twice again (with jz = h4,4) to yield 


—3.1219 0.7193 1.2718 —1.4630 
0.8637 1.8018 0.0868 —0.3916 
0 0.6770 = —1.2385 1.1642 
0 0 —0.0036 —0.5824 


and 


—3.0939 —0.6040 1.3771 1.2656 
—0.8305 1.8532 —0.3517 0.5050 
0 0.2000 —1.3114 —1.3478]° 
0 0 0.00003 —0.5888 


The (4,4) entry is an eigenvalue of A correct to four decimal places. 

This matrix happens to have a real eigenvalue. If it had not, Algorithm 4 could have been used 
with k = 2 to extract the complex eigenvalues in pairs. 
. For an example of an ill-conditioned eigenvalue (Fact 31) consider a matrix 


1 t 


A= 
0 l+e 


> 


where t is large or € is small or both. Since A is upper triangular, its eigenvalues are 1 and 1+. 
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Eigenvectors of A and A? associated with the eigenvalue 1 are 


1 1 
fl Li 


respectively. Since ||xl|, = 1, llyll, = /1 + #2/e2, and |y?x| = 1, the condition number of 
eigenvalue A = lisk = \/1+4 t?/e? © t/e. Thus if, for example, t = 10’ and « = 107’, we have 
«10. 

5. This example illustrates Fact 32 on the ill conditioning of eigenvectors associated with a tight cluster 
of eigenvalues. Given a positive number € that is as small as you please, the matrices 


2+€ 0 0 


A, = 0 2-—e€ 0 
0 0 1 
and 
€ 0 
A, = (Fi 2 0 
(0) 0 1 


both have eigenvalues 1, 2 + €, and 2 — ¢, and they are very close together: || Ay — A2||2 = V2e. 
However, unit eigenvectors associated with clustered eigenvalues 2 + € and 2 — € for A; are 


1 0 
e, = |0 and e,= /1], 
0 
while unit eigenvectors for Az are 
1 1 


Thus, the tiny perturbation of order € from A; to A; changes the eigenvectors completely; the 
eigenvectors are ill conditioned. In contrast the two-dimensional invariant subspace associated 
with the cluster 2 + €, 2 — € is Span(e),e) for both A; and Aj, and it is well conditioned. 


43.3 Sparse Matrix Techniques 


If the matrix A is large and sparse and just a few eigenvalues are needed, sparse matrix techniques are 
appropriate. Some examples of common tasks are: (1) find the few eigenvalues of largest modulus, (2) find 
the few eigenvalues with largest real part, and (3) find the few eigenvalues nearest some target value t. The 
corresponding eigenvectors might also be wanted. These tasks are normally accomplished by computing 
the low-dimensional invariant subspace associated with the desired eigenvalues. Then the information 
about the eigenvalues and eigenvectors is extracted from the invariant subspace. 

The most widely used method for the sparse unsymmetric eigenvalue problem is the implicitly restarted 
Arnoldi method, as implemented in ARPACK [LSY98], which is discussed in Chapter 76. A promising 
variant is the Krylov—Schur algorithm of Stewart [Ste01]. MATLAB’s sparse eigenvalue command “eigs” 
calls ARPACK. 
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Definitions: 


Given a subspace S of C”, a vector v € S is called a Ritz vector of A from S if there isa @ € C such that 
Av — 6v 1 S. The scalar 6 is the Ritz value associated with S. The pair (6, v) is a Ritz pair. 


Facts: 


1. [Wat02, § 6.1] Let vy, ... , Vin bea basis for a subspace S of C”, and let V = [v, «++ Vi]. Then S is 
invariant under A if and only if there isa B € C”*” such that AV = VB. 

2. [Wat02, § 6.1] If AV = VB, then the eigenvalues of B are eigenvalues of A. If x is an eigenvector 
of B associated with eigenvalue jz, then Vx is an eigenvector of A associated with pw. 

3. [Wat02, $6.4] Let v,, ..., Vin be an orthonormal basis of S, V = [v, --+Vm], and B = V*AV. 
Then the Ritz values of A associated with S are exactly the eigenvalues of B. If (0, x) is an eigenpair 
of B, then (0, Vx) is a Ritz pair of A, and conversely. 

4. If A is very large and sparse, it is essential to store A in a sparse data structure, in which only 
the nonzero entries of A are stored. One simple structure stores two integers n and nnz, which 
represent the dimension of the matrix and the number of nonzeros in the matrix, respectively. The 
matrix entries are stored in an array ent of length nnz, and the row and column indices are stored 
in two integer arrays of length nnz called row and col, respectively. For example, if the nonzero 
entry aj; is stored in ent(m), then this is indicated by setting row(m) = i and col(m) = j. The 
space needed to store a matrix in this data structure is proportional to nnz. 

5. Many operations that are routinely applied to dense matrices are impossible if the matrix is stored 
sparsely. Similarity transformations are out of the question because they quickly turn the zeros to 
nonzeros, transforming the sparse matrix to a full matrix. 

6. One operation that is always possible is to multiply the matrix by a vector. This requires one pass 
through the data structure, and the work is proportional to nnz. 


Algorithm 5. Sparse Matrix-Vector Multiply. Multiply A by x and store the result in y. 
y<0 
form = 1: nnz 
Ly(row(m)) <— y(row(m)) + ent(m) * x(col(m))] 
end 


7. Because the matrix-vector multiply is so easy, many sparse matrix methods access the matrix A in 
only this way. At each step, A is multiplied by one or several vectors, and this is the only way A is 
used. 

8. The following standard methodology is widely used. A starting vector v; is chosen, and the algorithm 
adds one vector per step, so that after j — 1 steps it has produced j orthonormal vectors vj, ... , 
v;.Let Vj = [w,...,vj] € C"/ and let S; = Span(V;) = Span(v,,...,vj;). The jth step uses 
information from Sj to produce vj+1. The Ritz values of A associated with S; are the eigenvalues 
of the j x j matrix Bj = Vj AVj. The Ritz pair (9, w) for which @ has the largest modulus is an 
estimate of the largest eigenvalue of A, and x = V;w is an estimate of the associated eigenvector. 
The residual r; = Ax — x0 gives an indication of the quality of the approximate eigenpair. 

9. Several methods use the residual r; to help decide on the next basis vector v;,. These methods 
typically use r; to determine another vector s;, which is then orthonormalized against v;, ... , Vj 
to produce vj+1. The choice s; = rj; leads to a method that is equivalent to the Arnoldi process. 
However, Arnoldi’s process should not be implemented this way in practice. (See Chapter 44.) The 
choices; = (D—6I tr j» where D is the diagonal part of A, gives Davidson’s method. The Jacobi— 
Davidson methods have more elaborate ways of choosing s;. (See [BDD00, § 7.12] for details.) 

10. Periodic purging is employed to keep the dimension of the active subspace from becoming too 
large. Given m vectors, the purging process keeps the most promising k-dimensional subspace of 
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Sm = Span(V,,) and discards the rest. Again, let B,, = V,“AV,,, and let B,, = UmTi,U;, be a 
unitary similarity transformation to upper triangular form. The Ritz values lie on the main diag- 
onal of T,,, and can be placed in any order. Place the k most promising Ritz values at the top. Let 
Vin = VinUm, and let V denote the n x k submatrix of V,, consisting of the first k columns. The 
columns of Vj are the vectors that are kept. 

11. After each purge, the algorithm can be continued from step k. Once the basis has been expanded 
back to m vectors, another purge can be carried out. After a number of cycles of expansion and 
purging, the invariant subspace associated with the desired eigenvalues will have been found. 

12. When purging is carried out in connection with the Arnoldi process, it is called an implicit restart, 
and there are some extra details. (See Chapter 44 and [Ste01]}). 

13. The implicitly restarted Arnoldi process is well suited for computing the eigenvalues on the 
periphery of the spectrum of A. Thus, it is good for computing the eigenvalues of maximum 
modulus or those of maximum or minimum real part. 

14. For computing interior eigenvalues, the shift-and-invert strategy is often helpful. Suppose the eigen- 
values nearest some target value t are sought. The matrix (A—t1)~' has the same eigenvectors as A, 
but the eigenvalues are different. IfA,,... ,A, are the eigenvalues of A,then(A;—t)7!... ,(Ay—T)7! 
are the eigenvalues of (A — tI)~!. The eigenvalues of (A — tI) ~! of largest modulus correspond to 
the eigenvalues of A closest to t. These can be computed by applying the implicitly restarted Arnoldi 
process to (A — tI)~!. This is feasible whenever operations of the type w < (A — tI)~!x can be 
performed efficiently. If a sparse decomposition A — tI = PLU can be computed, as described in 
Chapter 41, then that decomposition can be used to perform the operation w <— (A — tI)~!x by 
back solves. Ifthe L U factors take up too much space to fit into memory, this method cannot be used. 

15. Another option for solving (A — tI)w = xis to use an iterative method, as described in Chapter 
41. However, this is very computationally intensive, as the systems must be solved to high accuracy 
if the eigenvalues are to be computed accurately. 

16. The shift-and-invert strategy can also be applied to the generalized eigenvalue problem Av = A By. 
The implicitly restarted Arnoldi process is applied to the operator (A — 1 B)~'B to find eigenvalues 
near T. 

17. If the matrix is too large for the shift-and-invert strategy, Jacobi-Davidson methods can be con- 
sidered. These also require the iterative solution of linear systems. In this family of methods, 
inaccurate solution of the linear systems may slow convergence of the algorithm, but it will not 
cause the eigenvalues to be computed inaccurately. 

18. Arnoldi-based and Jacobi—Davidson algorithms are described in [BDD00]. A brief overview is given 
in [Wat02, § 6.4]. Balancing of sparse matrices is discussed in [BDD00, § 7.2]. 
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The implicitly restarted Arnoldi method (IRAM) [Sor92] is a variant of Arnoldi’s method for computing 
a selected subset of eigenvalues and corresponding eigenvectors for large matrices. Implicit restarting is a 
synthesis of the implicitly shifted QR iteration and the Arnoldi process that effectively limits the dimension 
of the Krylov subspace required to obtain good approximations to desired eigenvalues. The space is 
repeatedly expanded and contracted with each new Krylov subspace generated by an updated starting 
vector obtained by implicit application of a matrix polynomial to the old starting vector. This process is 
designed to filter out undesirable components in the starting vector in a way that enables convergence to 
the desired invariant subspace. This method has been implemented and is freely available as ARPACK. 
The MaTLaz® function eigs is based upon ARPACK. Use of this software is described in Chapter 76. 

In this article, all matrices, vectors, and scalars are complex and the algorithms are phrased in terms of 
complex arithmetic. However, when the matrix (or matrix pair) happens to be real then the computations 
may be organized so that only real arithmetic is required. Multiplication of a vector x by a scalar A is 
denoted by xd so that the eigenvector—eigenvalue relation is Ax = xd. This convention provides for direct 
generalizations to the more general invariant subspace relations AX = XH, where X is ann x k matrix 
and H isak x k matrix with k < n. More detailed discussion of all facts and definitions may be found in 
the overview article [Sor02]. 


44.1 Krylov Subspace Projection 


The classic power method is the simplest way to compute the dominant eigenvalue and correspond- 
ing eigenvector of a large matrix. Krylov subspace projection provides a way to extract additional 
eigen-information from the power method iteration by considering all possible linear combinations of the 
sequence of vectors produced by the power method. 
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Definitions: 


The best approximate eigenvectors and corresponding eigenvalues are extracted from the Krylov subspace 
K.(A, v) := span{v, Av, A’V,..., Ak ly}, 


The approximate eigenpairs are constructed through a Galerkin condition. An approximate eigenvector 
x € Sis called a Ritz vector with corresponding Ritz value @ if the Galerkin condition 


w* (Ax — x9) = 0, forall w € K,(A,v) 


is satisfied. 


Facts: [Sor92], [Sor02] 
1. Every w € K;, is of the form w = $(A)v; for some polynomial @ of degree less than k and 
Kj-1 EG Kj for j ie ee 
2. Ifa sequence of orthogonal bases Vi = [v1, V2,...,V%] has been constructed with K;, = range(V;) 
and VV; = I, then a new basis vector v;.41 is obtained by the projection formulas 


hy = V; Avi; 
f, = Avy — Vehx, 
Vi = f/Ifkll2- 
The vector hy is constructed to achieve V;‘f, = 0 so that vy; is a vector of unit length that is 
orthogonal to the columns of Vx. 
3. The columns of Vi4; = [Vi, Ve+1] provide an orthonormal basis for K,41(A, v1). 
4. The basis vectors are of the form vj; = @j-1(A)vi1, where $;-_1 is a polynomial of degree j — 1 for 


each j = 1,2,...,k41. 
5. This construction fails when f; = 0, but then 


AVk = Vi is 


where H; = V,*AV; = [hy, ho,..., hy] (with a slight abuse of notation). This “good breakdown” 
happens precisely when K;, is an invariant subspace of A. Hence, 0 (Hy) C o(A). 


44.2 The Arnoldi Factorization 


The projection formulas given above result in the fundamental Arnoldi method for constructing an 
orthonormal basis for Kx. 


Definitions: 


The relations between the matrix A, the basis matrix V; and the residual vector fj, may be concisely 
expressed as 


AV; = Vi Hk + fez, 


where Vj; € C”** has orthonormal columns, Vif, = 0, and H, = Vi AV; isak x k upper Hessenberg 
matrix with nonnegative subdiagonal elements. 
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The above expression shall be called a k-step Arnoldi factorization of A. When A is Hermitian, Hy 
will be real, symmetric, and tridiagonal and then the relation is called a k-step Lanczos factorization of 
A. The columns of V; are referred to as Arnoldi vectors or Lanczos vectors, respectively. The Hessenberg 
matrix Hy is called unreduced if all subdiagonal elements are nonzero. 


Facts: [Sor92], [Sor02] 
1. The explicit steps needed to form a k-step Arnoldi factorization are shown in Algorithm 1. 


Algorithm 1: k-step Arnoldi factorization. A square matrix A, a nonzero vector v and a 
positive integer k < n are input. 

Output is an n x k ortho-normal matrix V;, an upper Hessenberg matrix H; and a vector fy, 
such that AV; = Vi Hy + frez. 


vi, = v/Ilvllos 

w= Av); a) = VW; 

f; <— w- vq); 

VY <= [wis Mh <— [ar]; 

for j = 1,2,3,...k—1, 
Bj = Ej llos Vian s= f;/Bj3 
Vin — [Vj vjai]s 


w <— Avj4i5 

h< VWs 

fij41 <= Wie Vj4ih; 
Aj41 <— [Hj,h); 
end 


2. Ritz pairs satisfying the Galerkin condition (see Section 44.1) are derived from the eigenpairs of 
the small projected matrix H;. If Hyy = y@ with |ly||2 = 1, then the vector x = V;y is a vector of 
unit norm that satisfies 


|| Ax — xO |]2 = ||(AVk — Vie yll2 = |Bxexyl 


where Bx = ||fill2- 
3. If (x,@) is a Ritz pair constructed as shown in Fact 2, then 


6 =y" My = (Vy)*A( Vy) = x" Ax 


is always a Rayleigh quotient (assuming ||y||2 = 1). 

4. The Rayleigh quotient residual r(x) := Ax — x6 satisfies ||r(x)|l2 = |B,ezy|. When A is Hermitian, 
this relation provides computable rigorous bounds on the accuracy of the approximate eigenvalues 
[Par80]. When A is non-Hermitian, one needs additional sensitivity information. Nonnormality 
effects may corrupt the accuracy. In exact arithmetic, these Ritz pairs are eigenpairs of A whenever 
f, = 0. However, even with a very small residual these may be far from actual eigenvalues when A 
is highly nonnormal. 

5. The orthogonalization process is based upon the classical Gram—Schmidt (CGS) scheme. This 
process is notoriously unstable and will fail miserably in this application without modification. 


44-4 Handbook of Linear Algebra 


FIGURE 44.1 DGKS Correction. 


The iterative refinement technique proposed by Daniel, Gragg, Kaufman, and Stewart (DGKS) 
[DGK76] provides an excellent way to construct a vector fj; that is numerically orthogonal to 
Vj +1. It amounts to computing a correction 


c= Vi 


jritj4is finr — fj —- Ving h<oht+o 


just after computing fj, if necessary, i.e., when fj; is not sufficiently orthogonal to the columns 
of Vj41. This formulation is crucial to both accuracy and performance. It provides numerically 
orthogonal basis vectors and it may be implemented using the Level 2 BLAS operation -GEMV 
[DDH88]. This provides a significant performance advantage on virtually every platform from 
workstation to supercomputer. 

6. The modified Gram—Schmidt (MGS) process will generally fail to produce orthogonal vectors and 
cannot be implemented with Level 2 BLAS in this setting. ARPACK relies on a restarting scheme 
wherein the goal is to reach a state of dependence in order to obtain f, = 0. MGS is completely 
inappropriate for this situation, but the CGS with DGKS correction performs beautifully. 

7. Failure to maintain orthogonality leads to numerical difficulties in the Lanczos/Arnoldi pro- 
cess. Loss of orthogonality typically results in the presence of spurious copies of the approximate 
eigenvalue. 


Examples: 


1. Figure 44.1 illustrates how the DGKS mechanism works. When the vector w = Av is nearly in 
the range(V) then the projection Vh is possibly inaccurate, but vector = w — Vh is not close to 
range(V) and can be safely orthogonalized to compute the correction c accurately. The corrected 
vector f < f — Vewill be numerically orthogonal to the columns of V in almost all cases. Additional 
corrections might be necessary in very unusual cases. 


44.3 Restarting the Arnoldi Process 


The number of Arnoldi steps required to calculate eigenvalues of interest to a specified accuracy cannot be 
pre-determined. Usually, eigen-information of interest does not appear until k gets very large. In Figure 44.2 
the distribution in the complex plane of the Ritz values (shown in grey dots) is compared with the spectrum 
(shown as +s). The original matrix is a normally distributed random matrix of order 200 and the Ritz 
values are from a (k = 50)-step Arnoldi factorization. Eigenvalues at the extremes of the spectrum of A 
are clearly better approximated than the interior eigenvalues. 

For large problems, it is intractable to compute and store a numerically orthogonal basis set V; for large k. 
Storage requirements are O(n - k) and arithmetic costs are O(n - k”) flops to compute the basis vectors 
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FIGURE 44.2 Typical distribution of Ritz values. 


plus O(k?) flops to compute the eigensystem of Hy. Thus, restarting schemes have been developed that 
iteratively replace the starting vector v, with an “improved” starting vector v/ and then compute a new 
Arnoldi factorization of fixed length k to limit the costs. Beyond this, there is an interest in forcing f, = 0 
and, thus, producing an invariant subspace. However, this is useful only if the spectrum o (.H;) has the 
desired properties. 

The structure of f suggests the restarting strategy. The goal will be to iteratively force v; to be a linear 
combination of eigenvectors of interest. 


Facts: [Sor92], [Sor02] 
1. Ifv= pa qjyj where Aq; = qjA; and 


AV = VH + fez 


is a k-step Arnoldi factorization with unreduced H, then f = 0 and o(H) = {A),A2,..., Ax}. 

2. Since v; determines the subspace Kx, this vector must be constructed to select the eigenvalues of 
interest. The starting vector must be forced to become a linear combination of eigenvectors that 
span the desired invariant subspace. There is a necessary and sufficient condition for f to vanish 
that involves Schur vectors and does not require diagonalizability. 


44.4 Polynomial Restarting 


Polynomial restarting strategies replace v, by 
vi <— W(A)y, 
where y is a polynomial constructed to damp unwanted components from the starting vector. If vy) = 


Vj=1 ivi Where Aq; = qjA;, then 


vi =v(Aw = > qjvi¥(aj), 
j=l 
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where the polynomial w has also been normalized to give || vj ||, = 1. Motivated by the structure of f;,, the 
idea is to force the starting vector to be closer and closer to an invariant subspace by constructing y so 
that |y(A)| is as small as possible on a region containing the unwanted eigenvalues. 

An iteration is defined by repeatedly restarting until the updated Arnoldi factorization eventually 
contains the desired eigenspace. An explicit scheme for restarting was proposed by Saad in [Saa92]. One 
of the more successful choices is to use Chebyshev polynomials in order to damp unwanted eigenvector 
components. 


Definitions: 


The polynomial yf is sometimes called a filter polynomial, which may also be specified by its roots. The 
roots of the filter polynomial may also be referred to as shifts. This terminology refers to their usage 
in an implicitly shifted QR-iteration. One straightforward choice of shifts is to find the eigenvalues 6; 
of the projected matrix H and sort these into two sets according to a given criterion: the wanted set 
W = {6; : j = 1,2,...,k} and the unwanted set UU = {0; : j = k+1,k+2,...,k + p}. Then one 
specifies the polynomial y as the polynomial with these unwanted Ritz values as it roots. This choice of 
roots, called exact shifts, was suggested in [Sor92]. 


Facts: [Sor92], [Sor02] 


1. Morgan [Mor96] found a remarkable property of this strategy. If exact shifts are used to define 
v(t) = Tae — 6;) and if q; denotes a Ritz vector of unit length corresponding to 6;, then 
the Krylov space generated by vj = w(A)v; satisfies 


Km(A,vP) = Span{qi, q> see > ks Aq;> A’4; ce) AP qj}; 


for any j = 1,2,...,k. Thus, polynomial restarting with exact shifts will generate a new subspace 
that contains all of the possible choices of updated staring vector consisting of linear combinations 
of the wanted Ritz vectors. 

2. Exact shifts tend to perform remarkably well in practice and have been adopted as the shift selection 
of choice in ARPACK when no other information is available. However, there are many other 
possibilities such as the use of Leja points for certain containment regions or intervals [BCR96]. 


44.5 Implicit Restarting 


There are a number of schemes used to implement polynomial restarting. We shall focus on an implicit 
restarting scheme. 


Definitions: 


A straightforward way to implement polynomial restarting is to explicitly construct the starting vector 
vi = W(A)y, by applying y(A) through a sequence of matrix-vector products. This is called explicit 
restarting. A more efficient and numerically stable alternative is implicit restarting. This technique applies 
a sequence of implicitly shifted QR steps to an m-step Arnoldi or Lanczos factorization to obtain a truncated 
form of the implicitly shifted QR-iteration. 

On convergence, the IRAM iteration (see Algorithm 2) gives an orthonormal matrix V; and an upper 
Hessenberg matrix H; such that AV, © V, Hy. If Hy Qx = Qy Ry is a Shur decompositon of H;, then we 
call V, = V;Q, a Schur basis for the Krylov subspace K,(A, v,). Note that if AV, = V; Hy exactly, then 
Vj would form the leading k columns of a unitary matrix V and Ry would form the leading k x k block 
of an upper triangular matrix R, where AV = VR is a complete Schur decomposition. We refer to this as 
a partial Schur decomposition of A. 


The Implicitly Restarted Arnoldi Method 44-7 


end 


Algorithm 2: IRAM iteration 

Input is an n x k ortho-normal matrix V;, an upper Hessenberg matrix Hy and a vector fy 
such that AV; = Vi Hy + fez. 

Output is an n x k ortho-normal matrix V;, an upper triangular 

matrix H; such that AV; = Vi Hy. 


repeat until convergence, 


Beginning with the k-step factorization, 
apply p additional steps of the Arnoldi process 
to compute an m = k + p step Arnoldi factorization 
AVin = VinHin + fine, - 
Compute o(H,,) and select p shifts (11, (12, ...4p; 
Q = In 
for j = 1,2,..., p, 
Factor [Q;, Rj] = qr( An — 41); 
Q = QQj; 
end 
Be = Hn (k + 1,k); 0% = Q(m,k); 
fe <— vir Be + fnoKs 
Vi = VinQ(51:h); Ak <— An(1:k,1:k); 


Facts: [Sor92], [Sor02] 


1. 


Implicit restarting avoids numerical difficulties and storage problems normally associated with 
Arnoldi and Lanczos processes. The algorithm is capable of computing a few (k) eigenvalues with 
user specified features such as largest real part or largest magnitude using 2nk + O(k’) storage. The 
computed Schur basis vectors for the desired k-dimensional eigenspace are numerically orthogonal 
to working precision. 

Desired eigen-information from a high-dimensional Krylov space is continually compressed into 
a fixed size k-dimensional subspace through an implicitly shifted QR mechanism. An Arnoldi 
factorization of length m = k + p, 


AVin = Vin Ain + fine; 


is compressed to a factorization of length k that retains the eigen-information of interest. Then the 
factorization is expanded once more to m-steps and the compression process is repeated. 

QR steps are used to apply p linear polynomial factors A — 1; 1 implicitly to the starting vector v;. 
The first stage of this shift process results in 


AV* = VtH* + fne*,Q, 


where V,t = V,,Q, Hf = Q*HnQ, and Q = Q)Q)::- Q,. Each Q; is the orthogonal matrix 
associated with implicit application of the shift w~; = 6,,;. Since each of the matrices Q; is 
Hessenberg, it turns out that the first k—1 entries of the vector e*, Q are zero (ie.,e%,Q = [o e; ,q*]). 
Hence, the leading k columns remain in an Arnoldi relation and provide an updated k-step Arnoldi 
factorization 


AV, = V, Hy + foe, 


with an updated residual of the form f° = Vitex. B, + £0. Using this as a starting point, it is 
possible to apply p additional steps of the Arnoldi process to return to the original m-step form. 
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4, Virtually any explicit polynomial restarting scheme can be applied with implicit restarting, but 
considerable success has been obtained with exact shifts. Exact shifts result in H,’ having the k 
wanted Ritz values as its spectrum. As convergence takes place, the subdiagonals of H; tend to zero 
and the most desired eigenvalue approximations appear as eigenvalues of the leading k x k block 
of R as a partial Schur decomposition of A. The basis vectors V; tend to numerically orthogonal 
Schur vectors. 

5. The basic IRAM iteration is shown in Algorithm 2. 


Examples: 


1. The expansion and contraction process of the IRAM iteration is visualized in Figure 44.3. 
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FIGURE 44.3 Visualization of IRAM. 
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44.6 Convergence of IRAM 


IRAM converges linearly. An intuitive explanation follows. If v; is expressed as a linear combination of 
eigenvectors {q;} of A, then 


7 => ¥(A)y = S aut (Aj )¥j- 


j=l j=l 


Applying the same polynomial (i.e., using the same shifts) repeatedly for @ iterations will result in the j-th 
original expansion coefficient being attenuated by a factor 


( waj) ) ‘ 

wary) ° 

where the eigenvalues have been ordered according to decreasing values of | (A;)|. The leading k eigen- 
values become dominant in this expansion and the remaining eigenvalues become less and less significant 
as the iteration proceeds. Hence, the starting vector v, is forced into an invariant subspace as desired. 
The adaptive choice of y provided with the exact shift mechanism further enhances the isolation of the 
wanted components in this expansion. Hence, the wanted eigenvalues are approximated ever better as the 


iteration proceeds. Making this heuristic argument precise has turned out to be quite difficult. Some fairly 
sophisticated analysis is required to understand convergence of these methods. 


44.7 Convergence in Gap: Distance to a Subspace 


To fully discuss convergence we need some notion of nearness of subspaces. When nonnormality is present 
or when eigenvalues are clustered, the distance between the computed subspace and the desired subspace 
is a better measure of success than distance between eigenvalues. The subspaces carry uniquely defined 
Ritz values with them, but these can be very sensitive to perturbations in the nonnormal setting. 


Definitions: 


A notion of distance that is useful in our setting is the containment gap between the subspaces W and V : 


. |lw-v 
5(W, VY) := max fig le 
weW veV llwll2 


Note: 5(W, V) is the sine of the largest canonical angle between W and the closest subspace of V with the 
same dimension as W. 

In keeping with the terminology developed in [BER04] and [BES05], 4, shall be the invariant subspace 
of A associated with the so called “good” eigenvalues (the desired eigenvalues) and -Y; is the complementary 
subspace. P, and P; are the spectral projectors with respect to these spaces. 

It is desirable to have convergence in Gap for the Krylov method, meaning 


8(Km(A, vi), %y) > 0. 
Fundamental quantities required to study convergence. 
1. Minimal Polynomial for 4,: 
ag := minimal polynomial of A with respect to P,v,, 


which is the monic polynomial of least degree s.t. ag(A)P,vi = 0. 
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2: 


Facts: 
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Nonnormality constant &((2): The smallest positive number s.t. 
f(A) Tulle < «(2) maxl fO 


uniformly for all functions f analytic on Q. This constant and its historical origins are discussed 
in detail in [BER04]. 


. €-pseudospectrum of A: 


A,(A) = {2 €C: |I(2I — A)" |, > €"}- 


[BER04], [BES05] 


. Two fundamental convergence questions: 


* What is the gap 6(U,, Ky (A, v1)) as k increases? 

* How does 6(U,,K (A, V1)) depend on V7; = &(A)vj, and how can we optimize the asymptotic 
behavior? 

Key ingredients to convergence behavior are the nonnormality of A and the distribution of v, w. r. t. 


U,. The goal of restarting is to attain the unrestarted iteration performance, but within restricted 
subspace dimensions. 


. Convergence with no restarts: In [BES05], it is shown that 


BUg,Ke(A,vi)) <CoCy min max |1 — ag(z) p(2) 


> 


where the compact set 2, © C \ {2 contains all the good eigenvalues. 


Il (A)Poville 


<= max —————, Cy, := k(Qz)). 
vePn1 [|W (A)Pevill2 


0 


. Rate of convergence estimates are obtained from complex approximation theory. Construct con- 


formal map G taking the exterior of (2) to the exterior of the unit disk with G(oo) = oo and 
G’(oo) > 0. Define p := ( minj=),..,1 IGA I). Then (Gaier, Walsh) 


; ; 1/k 
limsup min max —p(z)| =p. 
ko00.) PEPK ZEQY 


ag(z) 


The image of {|z| = p~!} isa curve C := G~!({|z| = o~'}) around Q,. This critical curve passes 
through a good eigenvalue “closest to” 2, The curve contains at least one good eigenvalue, with all 
bad and no good eigenvalues in its interior. 


. Convergence with the exact shift strategy has not yet been fully analyzed. However, convergence 


rates have been established for restarts with asymptotically optimal points. These are the Fejér, 
Fekete, or Leja points for Q,. In [BESO5], computational experiments are shown that indicate that 
exact shifts behave very much like optimal points for certain regions bounded by pseudo-spectral 
level curves or lemniscates. 


. Let Wy, interpolate 1/a,(z) at the M restart shifts: 


b(Ug,K¢(A,V1)) < CoC, max |1 = Wr(z)ag(z)| < Co Cy C, eee 
ZELLp 


for any r > ¢ (see [Gai87], [FR89]). Here, ¥; = ®(A)vi, where ® is the aggregate restart poly- 
nomial (its roots are all the implicit restart shifts that have been applied). The subspace dimension 
is € = 2m, the restart degree is m, and the aggregate degree is M = vm. 
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44.8 The Generalized Eigenproblem 


In many applications, the generalized eigenproblem Ax = Mx, arises naturally. A typical setting is a finite 
element discretization of a continuous problem where the matrix M arises from inner products of basis 
functions. In this case, M is symmetric and positive (semi) definite, and for some algorithms this property 
is a necessary condition. Generally, algorithms are based upon transforming the generalized problem to a 
standard problem. 


44.9 Krylov Methods with Spectral Transformations 


Definitions: 


A very successful scheme for converting the generalized problem to a standard problem that is amenable 
to a Krylov or a subspace iteration method is to use the spectral transformation suggested by Ericsson 
and Ruhe [ER80], 


(A —oM)!Mx =xv. 


Facts: [Sor92], [Sor02] 


1. An eigenvector x of the spectral transformation is also an eigenvector of the original problem 
Ax = Mxi, with the corresponding eigenvalue given by A = o + +. 

2. There is generally rapid convergence to eigenvalues near the shift o because they are transformed to 
extremal well-separated eigenvalues. Perhaps an even more influential aspect of this transformation 
is that eigenvalues far from o are damped (mapped near zero). 

3. One strategy is to choose o to be a point in the complex plane that is near eigenvalues of interest 
and then compute the eigenvalues v of largest magnitude of the spectral trasformation matrix. It 
is not necessary to have o extremely close to an eigenvalue. This transformation together with the 
implicit restarting technique is usually adequate for computing a significant number of eigenvalues 
near o. 

4. Even when M = I, one generally must use the shift-invert spectral transformation to find interior 
eigenvalues. The extreme eigenvalues of the transformed operator A, are generally large and well 
separated from the rest of the spectrum. The eigenvalues v of largest magnitude will transform 
back to eigenvalues A of the original A that are in a disk about the point o. This is illustrated in 
Figure 44.4, where the + symbols are the eigenvalues of A and the circled ones are the computed 
eigenvalues in the disk (dashed circle) centered at the point o. 

5. With shift-invert, the Arnoldi process is applied to the matrix A, := (A — oM)~!M. Whenever a 
matrix-vector product w < A,Vv is required, the following steps are performed: 

*z= Mv, 

* Solve (A — 0 M)w = z for w. 

The matrix A — o M is factored initially with a sparse direct LU-decomposition or in a symmetric 
indefinite factorization and this single factorization is used repeatedly to apply the matrix operator 
Ag as required. 

6. The scheme is modified to preserve symmetry when A and M are both symmetric and M is 
positive (semi)definite. One can utilize a weighted M (semi) inner product in the Lanczos/Arnoldi 
process [ER80], [GLS94], [MS97]. This amounts to replacing the computation of h <— Viyiw and 


8; = |lfjll2 with h <— Vii Mw and Bj = , /£; Mf;, respectively, in the Arnoldi process described 
in Algorithm 1. 

7. The matrix operator A, is self-adjoint with respect to this (semi)inner product, 1e., (Agx,y) = 
(x, Acy) for all vectors x, y, where (w, v) := /w* Mv. This implies that the projected Hessenberg 
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FIGURE 44.4 Eigenvalues from shift-invert. 


matrix H is actually symmetric and tridiagonal and the standard three-term Lanczos recurrence is 
recovered with this inner product. 

8. There is a subtle aspect to this approach when M is singular. The most pathological case, when 
null(A) M null(M) # {0}, is not treated here. However, when M is singular there may be infinite 
eigenvalues of the pair (A, M) and the presence of these can introduce large perturbations to the 
computed Ritz values and vectors. To avoid these difficulties, a purging operation has been suggested 
by Ericsson and Ruhe [ER80]. Ifx = Vy with Hy = yd, then 


A,x = VHy + fe,y = x0 + fey. 


Replacing the x with the improved eigenvector approximation x < (x+ a fe; y) and renormalizing 
has the effect of purging undesirable components without requiring any additional matrix vector 
products with A,. 

9. The residual error of the purged vector x with respect to the original problem is 


leZyI 
|| Ax — MxAll2 = || Mfll2 oP ; 


where A = o + 1/6. Since |6| is usually quite large under the spectral transformation, this new 
residual is generally considerably smaller than the original. 
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45.1 Singular Value Decomposition 


Definitions: 


Given a complex matrix A having m rows and n columns, if o is a nonnegative scalar and u and v are 
nonzero m- and n-vectors, respectively, such that 


Av=ou and A*u=ov, 


then o is a singular value of A and u and vare corresponding left and right singular vectors, respectively. 
(For generality it is assumed that the matrices here are complex, although given these results, the analogs 
for real matrices are obvious.) 

If, fora given positive singular value, there are exactly t linearly independent corresponding right singular 
vectors and t linearly independent corresponding left singular vectors, the singular value has multiplicity 
t and the space spanned by the right (left) singular vectors is the corresponding right (left) singular space. 

Given a complex matrix A having m rows and n columns, the matrix product UX V* is a singular value 
decomposition for a given matrix A if 


* Uand V, respectively, have orthonormal columns. 
* & has nonnegative elements on its principal diagonal and zeros elsewhere. 
°* A=UXV*. 


Let p and q be the number of rows and columns of U. U ism x p, p < m,and V isn x q withg <n. 
There are three standard forms of the SVD. All have the ith diagonal value of X denoted o; and ordered 
as follows: 0; > 02 > +--+ > ox, andr is the index such that o, > 0 and either k = r or 0,4; = 0. 


1. p= mandq =n. The matrix D is m x n and has the same dimensions as A (see Figures 45.1 and 
45.2). 

2. p=q = min{m,n}.The matrix © is square (see Figures 45.3 and 45.4). 

3. If p =q =r, the matrix D is square. This form is called a reduced SVD and denoted is by US V* 
(see Figures 45.5 and 45.6). 
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FIGURE 45.1 The first form of the singular value decomposition where m > n. 


FIGURE 45.2 The first form of the singular value decomposition where m < n. 


FIGURE 45.3 The second form of the singular value decomposition where m > n. 


FIGURE 45.4 The second form of the singular value decomposition where m < n. 
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FIGURE 45.5 The third form of the singular value decomposition where r < n < m. 
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FIGURE 45.6 The third form of the singular value decomposition where r < m < n. 
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Facts: 


The results can be found in [GV96, pp. 70-79]. Additionally, see Chapter 5.6 for introductory material and 
examples of SVDs, Chapter 17 for additional information on singular value decomposition, Chapter 15 


for information on perturbations of singular values and vectors, and Section 39.9 for information about 
numerical rank. 


a 


10. 


11. 


12: 


. If UXV* is a singular value decomposition for a given matrix A, then the diagonal elements {o;} 


of © are singular values of A. The columns {u;}?_, of U and {v;}“_, of V are left and right singular 
vectors of A, respectively. 
If m > n, the first standard form of the SVD can be found as follows: 


(a) Let A*A = VAV* be an eigenvalue decomposition for the Hermitian, positive semidefinite 
n x n matrix A* A such that A is diagonal (with the diagonal entries in nonincreasing order) 
and V is unitary. 


(b) Let the m x n matrix & have zero off-diagonal elements and fori = 1,... ,n let o;, the ith 
diagonal element of ©, equal ¥/1;, the positive square root of the ith diagonal element of A. 


(c) Fori = 1,...,n, let the m x m matrix U have ith column, uj, equal to 1/0; Av; if o; 4 0. If 
o; = 0, let u; be of unit length and orthogonal to all uj; for j # i, then UXV* is a singular 
decomposition of A. 


Ifm <n the matrix A* has a singular value decomposition UX V* and VX? U* isa singular value 
decomposition for A. The diagonal elements of & are the square roots of the eigenvalues of AA*. 
The eigenvalues of A* A are those of AA* plus n — m zeros. The notation £7 rather than D* is 
used because in this case the two are identical and the transpose is more suggestive. All elements of 
& are real so that taking complex conjugates has no effect. 

The value of r, the number of nonzero singular values, is the rank of A. 

If A is real, then U and V (in addition to &) can be chosen real in any of the forms of the SVD. 
The range of A is exactly the subspace of C” spanned by the r columns of U that correspond to 
the positive singular values. 

In the first form, the null space of A is that subspace of C” spanned by the n —r columns of V that 
correspond to zero singular values. 

In reducing from the first form to the third (reduced) form, a basis for the null space of A has been 
discarded if columns of Vhave been deleted. A basis for the space orthogonal to the range of A (i.e., 
the null space of A* ) has been discarded if columns of U have been deleted. 

In the first standard form of the SVD, U and V are unitary. 

The second form can be obtained from the first form simply by deleting columns n + 1,... ,m of 
U and the corresponding rows of S, ifm > n, or by deleting columns m + 1,... ,n of V and the 
corresponding columns of S, ifm <n. If m #4 n, then only one of U and V is square and either 
UU* = I, or VV* = I, fails to hold. Both U*U = I, and V*V = Ip. 

The reduced (third) form can be obtained from the second form by taking only ther x r principle 
submatrix of X, and only the firstr columns of U and V . If Ais rank deficient (ie.,r < min{m, n}), 
then neither U nor V is square and neither U*U nor V*V is an identity matrix. 

If p < m, let U bean m x (m— p)matrix of columns that are mutually orthonormal to one another 
as well as to the columns of Uand define the m x m unitary matrix 


e=l0>U): 


Ifq <n,let V be ann x (n — q)matrix of columns that are mutually orthonormal to one another 
as well as to the columns of Vand define the n x n unitary matrix 


V=l[v Vv). 
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Let © be the m x n matrix 


Then 


A=USV*, AV =UZ"*, AX =VE'U", and AU =VE™. 
13. Let UX V*bea singular value decomposition for A, an m x n matrix of rank r. Then: 


(a) There are exactly r positive elements of © and they are the square roots of the r positive 
eigenvalues of A* A (and also AA*) with the corresponding multiplicities. 


(b) The columns of V are eigenvectors of A* A; more precisely, v; is a normalized eigenvector of 
A* A corresponding to the eigenvalue oF, and uj satisfies ojuj; = Avj. 


(c) Alternatively, the columns of U are eigenvectors of AA*; more precisely, uj is a normalized 


eigenvector of AA* corresponding to the eigenvalue oF, and v; satisfies 0jv; = A*uj. 


14. The singular value decomposition UX V* is not unique. If UX V* is a singular value decomposition, 
so is (—U)X(—V*). The singular values may be arranged in any order if the columns of singular 
vectors in U and V are reordered correspondingly. 

15. If the singular values are in nonincreasing order then the only option for the construction of © is 
the choice for its dimensions p and q and these must satisfyr < p< mandr <q <n. 

16. If A is square and if the singular values are ordered in a nonincreasing fashion, the matrix & is 
unique. 

17. Corresponding to a simple (i.e., nonrepeated) singular value oj, the left and right singular vectors, 
uj and vj, are unique up to scalar multiples of modulus one. That is, if uj and v; are singular 
vectors, then for any real value of 6 so are eu j and ely j» but no other vectors are singular vectors 
corresponding to oj. 

18. Corresponding to a repeated singular value, the associated left singular vectors u; and right sin- 
gular vectors v; may be selected in any fashion such that they span the proper subspace. Thus, if 


Uj... Uj, andv;,,... , Vj, are the left and right singular vectors corresponding to a singular value 

o; of multiplicity s, then so are ui» ... Uj, andv;,,...,Wj, ifand only if there exists ans x s uni- 

tary matrix Q such that [uj,>... ,u’j,] = [uj,,... ,uj,] Qand [My seed svi | =[vj,..-,Vj,] Q 
Examples: 


For examples illustrating SVD see Section 5.6. 


45.2 Algorithms for the Singular Value Decomposition 


Generally, algorithms for computing singular values are analogs of algorithms for computing eigenvalues 
of symmetric matrices. See Chapter 42 and Chapter 46 for additional information. The idea is always to 
find square roots of eigenvalues of A? A without actually computing A? A. As before, we assume the matrix 
A whose singular values or singular vectors we seek is m x n. All algorithms assume m > n; ifm <n, the 
algorithms may be applied to A’. To avoid undue complication, all algorithms will be presented as if the 
matrix is real. Nevertheless, each algorithm has an extension for complex matrices. Algorithm 1 is presented 
in three parts. It is analogous to the QR algorithm for symmetric matrices. The developments for it can 
be found in [GK65], [GK68], [BG69], and [GR70]. Algorithm 1a is a Householder reduction of a matrix 
to bidiagonal form. Algorithm Ic is a step to be used iteratively in Algorithm 1b. Algorithm 2 computes 
the singular values and singular vectors of a bidiagonal matrix to high relative accuracy [DK90], [Dem97]. 
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Algorithm 3 gives a “Squareroot-free” method to compute the singular values of a bidiagonal matrix to 
high relative accuracy — it is the method of choice when only singular values are desired [Rut54], [Rut90], 
[FP94], [PM00]. Algorithm 4 computes the singular values of an n x n bidiagonal matrix by the bisection 
method, which allows k singular values to be computed in O(kn) time. By specifying the input tolerance 
tol appropriately, Algorithm 4 can also compute the singular values to high relative accuracy. Algorithm 5 
computes the SVD of a bidiagonal by the divide and conquer method [GE95]. The most recent method, 
based on the method of multiple relatively robust representations (not presented here), is the fastest and 
allows computation of k singular values as well as the corresponding singular vectors of a bidiagonal matrix 
in O(kn) time [DP04a], [DP04b], [GLO3], [WLV05]. All of the above mentioned methods first reduce 
the matrix to bidiagonal form. The following algorithms iterate directly on the input matrix. Algorithms 
6 and 7 are analogous to the Jacobi method for symmetric matrices. Algorithm 6 — also known as the 
“one-sided Jacobi method for SVD” — can be found in [Hes58] and Algorithm 7 can be found in [Kog55] 
and [FH60]. Algorithm 7 begins with an orthogonal reduction of the m x n input matrix so that all the 
nonzeros lie in the upper m x n portion. (Although this algorithm was named biorthogonalization in 
[FH60], it is not the biorthogonalization found in certain iterative methods for solving linear equations.) 
Many of the algorithms require a tolerance € to control termination. It is suggested that ¢ be set to a small 
multiple of the unit round off precision €,. 


Algorithm 1a: Householder reduction to bidiagonal form: 

Input: m,n, A where A is m x n. 

Output: B, U, V so that B is upper bidiagonal, U and V are products of Householder matrices, 
and A= UBV!. 


1. B < A. (This step can be omitted if A is to be overwritten with B.) 
2 US Disa 
32° V = cas 
4. Fork =1,...,n 
a. Determine Householder matrix Q; with the property that: 
* Left multiplication by Q; leaves components 1,... ,k — 1 unaltered, and 
0 0 
0 0 
* O Dkk = Dkk , where s = +,/ 57 b?,. 
bik Ss ixk ? 
Dik 0 
Dink 0 
b. B <— Q,B. 
c. U — UQ,. 


d. Ifk < n— 2, determine Householder matrix P+, with the property that: 


* Right multiplication by P;,; leaves components 1,... , k unaltered, and 
© [0 --- 0 dee bint bento +++ Din] Peer = [0--- 0 Dap s 0-+- Of, 


where s = +4/ 77 p41 Dg ;- 


e. B< BPyy. 
£-V Pav. 
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Algorithm 1b: Golub-Reinsch SVD: 
Input: m,n, A where A is m x n. 
Output: &, U, V so that & is diagonal, U and V have orthonormal columns, U is m x n, V is 
nxn,andA=UZXV’!. 
1. Apply Algorithm 1a to obtain B, U, V so that B is upper bidiagonal, U and V are products of 
Householder matrices, and A = UBV". 
2. Repeat: 
a. Ifforanyi=1,...,n—1, 


biiai| <& (lbii| + |Bia1i4i|) . set big = 0. 
b. Determine the smallest p and the largest q so that B can be blocked as 


By, 860 0 Pp 
B= 0 Bop 0 n—-p—-q 
0 0 B33 q 


where B33 is diagonal and B, has no zero superdiagonal entry. 
c. Ifq =n, set Y = the diagonal portion of B STOP. 
d. Iffori = p+1,...,n—q —1,);; = 0, then 
Apply Givens rotations so that b;;,; = 0 and Bp, is still 
upper bidiagonal. (For details, see [GL96, p. 454].) 
else 
Apply Algorithm Ic to n, B, U, V, p,q. 


Algorithm 1c: Golub—Kahan SVD step: 

Input: n, B, Q, P, p,q where B is n x n and upper bidiagonal, Q and P have orthogonal 
columns, and A = QBP?. 

Output: B, Q, P sothat B is upper bidiagonal, A= QBP T, Qand P have orthogonal columns, 
and the output B has smaller off-diagonal elements than the input B. In storage, B, Q, and P 
are overwritten. 


1. Let By be the diagonal block of B with row and column indices p+ 1,...,n—q. 
2. Set C = lower, right 2 x 2 submatrix of BJ, By). 

3. Obtain eigenvalues A), A2 of C. Set 4p = whichever of A,, A2 that is closer to cz. 
4,k= ptla= bey —p,p= be Ok R41 

5. Fork=p+t+l,...,n-q-1 


a. Determine c = cos(@) and s = sin(@) with the property that: 
cos} 5 5 
[a eau oh = [va +B 0}. 


b. B <— BRxx+1(c,5) where Ry~41(c, 5) is the Givens rotation matrix that acts on columns k 
and k + 1 during right multiplication. 

c. P — PRxxEi(c,5). 

d.a= bk, B = betLk: 

e. Determine c = cos(@) ands = sin(@) with the property that: 


Vet P 


0 


f. B<— Rxggail(c, —s)B, where Ry441(c, —s) is the Givens rotation matrix that acts on rows k 
and k + 1 during left multiplication. 

8 Q<— QRuksilcss). 

h. ifk<n—- q- la = baka B = De k42- 
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Algorithm 2a: High Relative Accuracy Bidiagonal SVD: 

Input: n, B where B is ann x n upper bidiagonal matrix. 

Output: & is an n x n diagonal matrix, U and V are orthogonal n x n matrices, and B = 
ep ph ge 


1. Compute o to be a reliable underestimate of Omin(B) (for details, see [DK90]). 
2. Compute = max; (b;;;, bjj41). 
3. Repeat: 
a. Foralli = 1,...,n— 1, set bjj+1 = 0 ifa relative convergence criterion is met (see [DK90] 
for details). 


b. Determine the smallest p and largest q so that B can be blocked as 


By, 60 0 P 
B= 0 By» 0 n—-p—q 
0 0 B33 q 


where B33 is diagonal and B», has no zero superdiagonal entry. 
c. Ifq =n, set D = the diagonal portion of B. STOP. 
d. Iffori = p+1,...,n—q—1, bj; = 0, then 
Apply Givens rotations so that b;;4; = 0 and Bp, is still 
upper bidiagonal. (For details, see [GV96, p. 454].) 
else 
Apply Algorithm 2b with n, B, U, V, p,q,0, @ as inputs. 


Algorithm 2b: Demmel-Kahan SVD step: 

Input: n, B, Q, P, p,q,0,o0 where B is n x n and upper bidiagonal, Q and P have orthogonal 
columns such that A = QBP!,@ ~ ||B|| anda is an underestimate of omin(B). 

Output: B, Q, P sothat B is upper bidiagonal, A= QBP T Qand P have orthogonal columns, 
and the output B has smaller off-diagonal elements than the input B. In storage, B, Q, and P 


are overwritten. 


1. Let By be the diagonal block of B with row and column indices p+ 1,...,n— q. 
2. If tol*o < ea, then 
ase Se =, 
b. Fork = p+1,n—q-1 
8 = ches B = de K+. 
* Determine c and s with the property that: 


la Bp] e i =[r 0], wherer = \/a? + B?. 


*Ifk Apt ibe =s'r. 
© P — PRxxaiilc,s), where Ryx41(c, 5) is the Givens rotation matrix that acts on columns 
k and k + 1 during right multiplication. 


sa=c'r,B = sberiksi- 


(continued) 
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Algorithm 2b: Demmel—Kahan SVD step: (Continued) 


* Determine c’ and s’ with the property that: 


JOtP 


0 


© Q< QRex+il(c, —5), where Ri441(c, —s) is the Givens rotation matrix that acts on rows 
k and k + 1 during left multiplication. 


* ben = Va? + B?. 


Cc. Bn—g—1n—q = (Bn—qn—q©)S'5 Bn—g.n—a = (Bp—gn—gc)e’. 
Else 


d. Apply Algorithm 1c to n, B, Q, P, p,q. 


Algorithm 3a: High Relative Accuracy Bidiagonal Singular Values: 
Input: n, B where B is ann x n upper bidiagonal matrix. 
Output: X is an n x n diagonal matrix containing the singular values of B. 


1. Square the diagonal and off-diagonal elements of B to form the arrays s and e, respectively, i.e., 


. 2 2 2 
fori =1,...,n—1,s; = bj;,e; = b7;,,, end for s, = b;,,. 
2. Repeat: 
a. For alli = 1,...,n—1, set e; = 0 ifa relative convergence criterion is met (see [PM00] for 
details). 


b. Determine the smallest p and largest q so that B can be blocked as 


By, =O 0 P 
B= 0 Bop 0 n—-p—-q 
0 0 B33 q 


where B33 is diagonal and B>» has no zero superdiagonal entry. 


c. Ifq =n,set Xo = \/diag(s). STOP. 
d. Iffori = p+1,...,n—q—1,s; =O then 
Apply Givens rotations so that e; = 0 and By» is still 
upper bidiagonal. (For details, see [GV96, p. 454].) 
else 
Apply Algorithm 3b with inputs 1, s, e. 


Algorithm 3b: Differential quotient-difference (dqds) step: 

Input: n, s, e wheres and eare the squares of the diagonal and superdiagonal entries, respectively, 
of ann x n upper bidiagonal matrix. 

Output: s and e are overwritten on output. 


1. Choose jz by using a suitable shift strategy. The shift 2 should be smaller than o;in(B)?. See 
[FP94,PMO00] for details. 
2.d= S$; — LP. 
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Algorithm 3b: Differential quotient-difference (dqds) step: (Continued) 
1. Fork =1,...,n—1 
a. Sspe=adte,. 
b. t = Sx4i/Ssk. 
C. ee = ext. 
d. d=dt— uw. 
e. Ifd < 0, go to step 1. 
2. sn =d. 


Algorithm 4a: Bidiagonal Singular Values by Bisection: 

Input: n, B,a, B, tol where n x n is a bidiagonal matrix, [q, B) is the input interval, and tol is 
the tolerance for the desired accuracy of the singular values. 

Output: w is the output array containing the singular values of B that lie in [q, 6). 


1. Ny = Negcount(n, B, a). 

. ng = Negcount(n, B, B). 

. Ifng = np, there are no singular values in [a, 8). STOP. 
. Put [a, ng, B, ng] onto Worklist. 

. While Worklist is not empty do 


ne WN 


a. Remove [low, now, UP, Mup| from Worklist. 
b. mid = (low + up)/2. 
c. If (up — low < tol), then 
* Fori = Now + 1,nup, wi — ng) = mid; 
Else 
° Mmia = Negcount(n, B, mid). 
* IfMmid > Now then 
Put [low, now, mid, Nig] onto Worklist. 
* Iftup > Mmia then 
Put [mid, Mnids UP; Nup] onto Worklist. 


Algorithm 4b: Negcount (n, B, 1): 
Input: The n x n bidiagonal matrix B and a number ju. 
Output: Negcount, i.e., the number of singular values smaller than jz is returned. 


lt=-—wp. 

2. Fork =1,...,n—1 
d=bi, +t. 
If (d < 0) then Negcount = Negcount + 1 
t=t* (bi,4,/d) — pw. 

End for 
3, d=b2,, +t. 
4. If(d < 0), then Negcount = Negcount + 1. 
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Algorithm 5: DC_SVD(n, B, &, U, V): Divide and Conquer Bidiagonal SVD: 

Input: n, B where B is an (n + 1) x n lower bidiagonal matrix. 

Output: © is ann x n diagonal matrix, U is an (n + 1) x (a + 1) orthogonal matrix, V is an 
orthogonal n x n matrix, so that B = UX vl. 


1. Ifn < no, then apply Algorithm 1b with inputs n + 1,n, B to get outputs &, U, V. 
Else 


By, ape, O 
Let B = , where k = n/2. 
O pre; By 


a. Call DC_SVD(k — 1, Bi, ©1,Ui, W)). 
b. Call DC_SVD(n — k, By, D2, Ur, Wo). 


a 


. Partition U; = (Q; qj), fori = 1,2, where q; is a column vector. 


o 


Extract 1) = Qle,Ay = qrez,l, = Qhey,Ar = her. 


e. Partition B as 


To 0 0 0 WwW, 0 T 
Cc 0 —-s al, & 0 M 
_ (6% Qi oi kh dh i me 8 465 2g) si 
sogz2 90 Qza  Coqz Bln 0 Xe 0 0 W 0 


0 


Oo 
oO 


where ro = y/ (adi)? + (BeAz)*; Co = aA1/T0s S50 = Bear/To- 


f. Compute the singular values of M by solving the secular equation 


n 2 
Z 
fw)=1+)> 5+, =0, 
ae dz — w? 


and denote the computed singular values by #1, W2,... Wh. 


g. Fori = 1,...,n, compute 


h. Fori = 1,...,n, compute the singular vectors 

( : 

uj = 7 we > 5 Teer a 
d; — ww? ra oa 5) 

( ‘ dyZz i ss (dy24)? 

vieo= a) AQ SD =) 2 
d; Twi Wi » 2 (dj ~ a 
and let U = [uy,... ,u,], VV = [v,... Vn]. 


diag(w1,W2,... 
0 


i. Return © = ( ae ,U + (QU q),V< WV. 


Computation of the Singular Value Decomposition 45-11 


Algorithm 6: Biorthogonalization SVD: 

Input: m,n, A where A is m x n. 

Output: &, U, V so that & is diagonal, U and V have orthonormal columns, U is m x n, V is 
nxn,andA=UZXV’". 


1. U < A. (This step can be omitted if A is to be overwritten with U.) 
2. Vi Tascas 


. Set N? = (3 > ui) = 0, and first = true. 


i=l j=l 


Qo 


4. Repeat until s!/* < e?N? and first = false. 
a. Sets = 0 and first = false. 
b. Fori=1,...,n—1. 
i. Forj =i+1,...,n 


m 2 
*s<st (= mist) ‘ 
k=1 


* Determine dj, d), c = cos(@), and s = sin(@) such that: 


* U — UR;,,(c,s) where R;,;(c,s) is the Givens rotation matrix that acts on columns i 
and j during right multiplication. 
“Va VRi,j (C5). 


5. Fori=1,...,m: 


a. oj) = 4/0 Ui; 
k=1 
b. U << Ux"! 


Algorithm 7: Jacobi Rotation SVD: 

Input: m,n, A where A is m x n. 

Output: &, U, V so that & is diagonal, U and V have orthonormal columns, U is m x n, V is 
nxn,andA=UZXV!. 


1. B < A. (This step can be omitted if A is to be overwritten with B.) 
pee © [ers Deoceyor 
35° V =HTcn: 
4, Ifm > n, compute the QR factorization of B using Householder matrices so that B < QA, 

where B is upper triangular, and let U < UQ. (See A6 for details.) 

aan ee 
5. Set N° = a oS bj iss = 0, and first = true. 
i=1 j=l 

6. Repeat until s < 6? N? and first = false. 


a. Sets = 0 and first = false. 
b. Fori=1,...,n—1 


(continued) 
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Algorithm 7: Jacobi Rotation SVD: (Continued) 


i. Forj =i+1,...,n: 
ssasthi, +i. 
* Determine d), d),c = cos(@) and s = sin(g) with the property that d; and d, are 


positive and 
c —S bij bij é $ _ d, 0 
Ss Cc bj bj, —$ ¢é x 0 dy : 


* B <— R,j(c,s)BRi,;(¢, —$) where R;,;(c, s) is the Givens rotation matrix that acts 
on rows i and j during left multiplication and R;,;(¢, —$) is the Givens rotation 
matrix that acts on columns i and j during right multiplication. 

“U< UR;,j(c,s). 

“Va VRi,j(¢, 5). 

7. Set X to the diagonal portion of B. 
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To compute the eigenvalues and singular values to high relative accuracy means to have a guaranteed 
number of accurate digits in all computed approximate values. If 1; is the computed approximation of 
Aj, then the desirable high relative accuracy means that |A; — hil < e |Aj|, whereO<¢ <1 independent 
of the ratio |A;|/ max; |A;|. This is not always possible. The proper course of action is to first determine 
classes of matrices and classes of perturbations under which the eigenvalues (singular values) undergo 
only small relative changes. This means that the development of highly accurate algorithms is determined 
by the framework established by the perturbation theory. 

Input to the perturbation theory is a perturbation matrix whose size is usually measured in a matrix 
norm. This may not be always adequate in numerical solutions of real world problems. If numerical 
information stored in the matrix represents a physical system, then choosing different physical units 
can give differently scaled rows and columns of the matrix, but representing the same physical system. 
Different scalings may be present because matrix entries represent quantities of different physical nature. 
It is possible that the smallest matrix entries are determined as accurately as the biggest ones. Choosing the 
most appropriate scaling from the application’s point of view can be difficult task. It is then desirable that 
simple change of reference units do not cause instabilities in numerical algorithms because changing the 
description (scaling the data matrix) does not change the underlying physical system. This issue is often 
overlooked or ignored in numerical computations and it can be the cause of incorrectly computed and 
misinterpreted results with serious consequences. 
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This chapter describes algorithms for computation of the singular values and the eigenvalues of sym- 
metric matrices to high relative accuracy. Accurate computation of eigenvectors and singular vectors is not 
considered. Singular values are discussed in sections 46.1 to 46.3; algorithms for computation of singular 
values are also given in Chapter 45.2. 

Algorithms for computing the eigenvalues of symmetric matrices are given in Chapter 42. Sections 45.4 
to 45.5 discuss numerical issues concerning the accuracy of the computed approximate eigenvalues, divided 
into the cases positive definite and symmetric indefinite (a negative definite matrix A is handled by applying 
methods for positive definite matrices to — A). 

For symmetric H (which may contain initial uncertainty), perturbation theory determines whether 
or not it is numerically feasible to compute even the smallest eigenvalues with high relative accuracy. 
State of the art perturbation theory, which is a necessary prerequisite for algorithmic development, is 
given in Chapter 15. The insights from the perturbation theory are then capitalized in the development 
of numerical algorithms capable of achieving optimal theoretical accuracy. Unlike in the case of standard 
accuracy, positive definite and indefinite matrices are analyzed separately. 

All matrices are assumed to be over the real field R. Additional relevant preparatory results can be found 
in Chapter 8 and Chapter 17. 


46.1 Accurate SVD and One-Sided Jacobi SVD Algorithm 


Numerical computation of the SVD inevitably means computation with errors. How many digits in the 
computed singular values are provably accurate is answered by perturbation theory adapted to a particular 
algorithm. For dense matrices with no additional structure, the Jacobi SVD algorithm is proven to be 
more accurate than any other method that first bidiagonalizes the matrix. The simplest form is the one- 
sided Jacobi SVD introduced by Hestenes [Hes58]. It represents an implicit form of the classical Jacobi 
algorithm [Jac46] for symmetric matrices, with properly adjusted stopping criterion. Basic properties of 
classical Jacobi algorithm are listed in Section 42.7. Detailed analysis and implementation details can be 
found in [DV92], [Drm94], [Mat96], and [Drm97]. 


Definitions: 


Two vectors x, y € R” are numerically orthogonal if |x’ y| < ¢||x||2|ly||2, where ¢ > 0 is at most round-off 
unit € times a moderate function of m. 

The square matrix U is a numerically orthogonal matrix if each pair of its columns is numerically 
orthogonal. 

A numerical algorithm that computes approximations U ~ U, V © V and % © » of the SVD 
A = UXV' of A is backward stable if U, V are numerically orthogonal and UV! = A+ 5A, with 
backward error 5A small compared to A. 


Facts: 


1. There is no loss of generality in considering real tall matrices, i.e., m > n. In case m < n, consider 
A’ with the SVD A? = VETU!. 

2. Let U + U, V © V, © & © be the approximations of the SVD of A = UXV", computed by 
a backward stable algorithm as A+ 5A = UXV’". Since the orthogonality of U and V cannot 
be guaranteed, the product UXV! in general does not represent an SVD. However, there exist 
orthogonal U close to U andan orthogonal V, close to V, such that (I + E,)(A+65A)(1 + E>) = 
USV", where E 1» Ep, which represent departure from orthogonality of U, V, are small in norm. 

3. Different algorithms produce differently structured 5 A. Consider the singular values 0 > --- > oO} 
andé, >--- > 6, of Aand A+ 6A, respectively. 


° If ||SAll2 < e|| All2 and without any additional structure of 5 A (6 A small in norm), then the best 
; ; ; : Oj — Oj 
error bound in the singular values is max |o; — &;| < || All2, Le., max Wey < k)(A) €. 
l<i<n lsi<n Oj 
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* [DV92] Let, for alli, the ith column 6a; of 6A satisfy ||5aj||2 < €|la;||2, where a; is the ith column 
of A. (5A is column-wise small perturbation.) Then A+ 6A = (B + 6B)D, where A = BD, 


D = diag(|lai|l2,.-- 5 llan|l2), and ||SBl|z < /ne. Let A have full column rank and let p be the 
rank of 6A. Then max ar Sil |5BB"' ||, < /pel| B' [2 
<i<nG; 


* [vdS69] Bio < k2(B) < Vn min k2(AS) < /nk2(A). This implies that a numerical algo- 
=diag 


rithm with column-wise small backward error 6A computes more accurate singular values than 
an algorithm with backward error that is only small in norm. 


4. In the process of computation of the SVD of A, an algorithm can produce intermediate matrix 
with singular values highly sensitive even to smallest entry-wise rounding errors. 

5. The matrix A may have initial uncertainty on input and, in fact, A = Ag+6Ao, where Ao is 
the true unknown data matrix and 6 Ap the initial error already present in A. If 5A generated 
by the algorithm is comparable with 5 Ao, the computed SVD is as accurate as warranted by the 
data. 

6. [Hes58] If H = AA, then the classical Jacobi algorithm can be applied to H implicitly. The 
one-sided (or implicit) Jacobi SVD method starts with general m x n matrix A® = A and it 
generates the sequence A“+) = AV), where the matrix V“ is the plane rotation as in the 
classical symmetric Jacobi method for the matrix H® = (A®)TA®, Only H [{ix, jx} ] is needed 
to determine V“), where (ix, jx) is the pivot pair determined by pivot strategy. 

7. In the one-sided Jacobi SVD algorithm applied to A € R”*", m > n, A tends to UX, 
with diagonal 2 = diag(oi,...,0,) carrying the singular values. If A has full column rank, 
then the columns of U are the n corresponding left singular vectors. If rank(A) < n, then for 
each o; = 0 the ith column of U is zero. The m-rank( A) left singular vectors from the orthogonal 
complement of range( A) cannot be computed using one-sided Jacobi SVD. The accumulated prod- 
uct VV ... converges to orthogonal matrix V of right singular vectors. 

8. Simple implementation of Jacobi rotation in the one-sided Jacobi SVD algorithm (one-sided Jacobi 
rotation) is given in Algorithm 1. At any moment in the algorithm, d),... ,d, contain the squared 
Euclidean norms of the columns of current A“, and € stores the Euclidean inner product of the 
pivot columns in the kth step. 


Algorithm 1: One-sided Jacobi rotation 
ROTATE (Aim Ain, j> d; > dj, é, [Visi Vismj }) 


F rs d; — dj ; sign(v) 1 
: = id 3o= 3S SECS 
2-& lo) + J/14 02 V14+? 
Cc Ss 
2: [ Alem Ai:m,j] = [Atm Aim] } 
—s Cc 


3: dj =d;—t-&;dj;=dj+t-&; 
4: if V is wanted, then 


5: [Vieni Vien, j] —_ [Vieni Viznj | 


Cc Ss 
—s C¢ 


In case of cancellation in computation of the smaller of dj, d;, the value is refreshed by explicit 
computation of the squared norm of the corresponding column. 
9. [dR89], [DV92] Numerical convergence of the one-sided Jacobi SVD algorithm (Algorithm 2) 
\(A®) yr aw | 


limi lim, j 


6: end if 


is declared at step k if max ——__—_ 
AI | Atm allall Aton, jll2 


limi lim, j 


< ¢ & me. This stopping criterion guarantees 


that computed approximation A“ of A“ can be written as US, where the columns of U are 
numerically orthogonal, and & is diagonal. 
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Algorithm 2: One-sided Jacobi SVD (de Rijk’s row-cyclic pivoting) 
(U, x, [V]) = SVDO(A) 
€¢ =me;p=n(n—1)/2;s = 0; convergence = false ; 
if V is wanted, then initialize V = I, end if 
fori = 1tondod; = Alm Alimi end for; 
repeat 
s=s+1;p=0; 
fori = 1ton—1do 
find index ig such that dj, = max; <<, de; swap(d;, dj,) ; 
swap(Atin,is Ati )3 SWAP(Virm,is Vizmio) 5 
for j =i+1tondo 
g= Alm Alm j3 
if |&| > ¢\/djd; then % apply Jacobi rotation 
call ROTATE(At-m,j5 Aiim,j> d;, dj, é, [Vismjis Vim, j]) 5 
else p = p+ 1endif 
end for 
end for 
if p = p, then convergence=true; go to > end if 
until s > 30 
e if convergence, then % numerical orthogonality reached 
for i=1 to n do dj; = Vd; 3 Utinj = Aimi Z| end for 
else 
Error: Numerical convergence did not occur after 30 sweeps. 
end if 


[DV92], [Mat96], [Drm97] Let A”, k = 0,1,2,... be the matrices computed by Algorithm 2 in 
floating point arithmetic. Write each A® as A® = B™ D™, with diagonals D™ and B“™ with 
columns of unit Euclidean norms. Then 


© AD = (AM 4 5A) V, where V is the exact plane rotation transforming pivot columns, 
and 5A) is zero except in the i,th and the j,th columns. 

© AX 458A = (BY +5B) D®, with |}5B™ || p < cpe, where c; isa small factor (e.g., ck < 20) 
that depends on the implementation of the rotation. 

* The above holds as long as the Euclidean norms of the ith and the j,th columns of A“ do not 
underflow or overflow. It holds even if the computed angle is so small that the computed value of 
tan ¢, underflows (gradually or flushed to zero). This, however, requires special implementation 
of the Jacobi rotation. 


* Ifall matrices AY, j =0,...,k — 1 are nonsingular, then for alli = 1,... ,n, 
k-1 i k-1 
o;(A®) 
max {0, [Ja nj) < <][[a+n)), 
jc0 ai(A) ~ 55 


where yj = SAP (AM) I < cjéll (B')*|],, and o;(-) stands for the ith largest singular value 
of a matrix. The accuracy of Algorithm 2 is determined by 8 = maxgso ||(B (K))t ||,. Itis observed 
in practice that 6 never exceeds || B* ||, too much. 


* If A initially contains column-wise small uncertainty and if 6 /|| B* ||, is moderate, then Algorithm 
2 computes as accurate approximations of the singular values as warranted by the data. 


Examples: 


1. 


Using orthogonal factorizations in finite precision arithmetic does not guarantee high relative 
accuracy in SVD computation. Bidiagonalization, as a preprocessing step in state-of-the-art SVD 
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algorithms, is an example. We use MATLAB with roundoff € = eps © 2.22. 107!*. Let & = 10/e. 
Then floating point bidiagonalization of 


1 1 fl l a 0 la 0 
A=|0 1 €&| yields AV=]0 B Bl, AP=]0 y y|, 
OH. . <e 0 |B] B 0 0 0 


a © 1.4142135e+0, B © 3.1845258et16, y ~ 4.5035996e+16. The matrices A“) and A®) are 
computed using Givens plane rotations at the positions indicated by| - |. The computed matrix A‘ 


entry-wise approximates with high relative accuracy the matrix A) from the exact computation 
(elimination of A13). However, A and A") are nonsingular and AY is singular. The smallest 
singular value of A is lost — A“ carries no information about 03. Transformation from A to 
A®) is again perfectly entry-wise accurate. But, even exact SVD of the bidiagonal A” cannot restore 
the information lost in the process of bidiagonalization. (See Fact 4 above.) 

On the other hand, let A denote the matrix obtained from A by perturbing Aj; to zero. This 
changes the third column a; of A by ||da3||2 < (€/14)||a3||2 and causes, at most, €/5 relative pertur- 
bation in the singular values. (See Fact 3 above. Here, A = BD with x(B) < 2.) If we apply Givens 
rotation from the left to annihilate the (3, 2) entry in A, the introduced perturbation is column- 


wise small. Also, the largest singular value decouples; the computed matrix is A?) = E , Qa. 


46.2 Preconditioned Jacobi SVD Algorithm 


The accuracy properties and the run-time efficiency of the one-sided Jacobi SVD algorithm can be enhanced 
using appropriate preprocessing and preconditioning. More details can be found in [Drm94], [Drm99], 
[DV05a], and [DV05b]. 


Definitions: 
. Boa R . 
QR factorization with column pivoting of A € R”*” is the factorization AI] = Q | i , where IT is a 


permutation matrix, Q is orthogonal, and R is n x n upper triangular. 

The QR factorization with Businger-Golub pivoting chooses I to guarantee that r7, > >~/_, rj for 
all j =k,...,nandk=1,...,n. 

The QR factorization with Powell—Reid’s complete (row and column) pivoting computes the QR 


factorization using row and column permutations, I, ATI, = Q 0? where IT, is as in Businger-Golub 


pivoting and I, enhances numerical stability in case of A with differently scaled rows. 
QR factorization with pivoting of A is rank revealing if small singular values of A are revealed by 
correspondingly small diagonal entries of R. 


Facts: 
1. [Drm94], [Hig96]. Let the QR factorization A = Q of A € R”*", m > n, be computed using 


the Givens or the Householder algorithm in the IEEE floating point arithmetic with rounding 
relative error € < 107’. Let the computed approximations of Q and R be Q and R, respectively. 
Then there exist an orthogonal matrix Q and a backward perturbation 6A such that 


. |R . ‘ 
A + bA= Q S| 5) || Quimi = Qi:mill2 < €qr> 6 Aten lla < Eqr || Atm, ll2 
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holds for alli = 1,... ,n, with €,, < O(mn)e. This remains true if the factorization is computed 
with pivoting. It suffices to assume that the matrix A is already prepermuted on input, and that the 
factorization itself is performed without pivoting. 


. [Drm94], [Mat96], [Drm99] Let A = BD, where D is diagonal and B has unit columns in 


Euclidean norm. 


* If || B"||2 is moderate, then the computed R allows approximation of the singular values of A to 
high relative accuracy. If R is computed with Businger-Golub pivoting, then Algorithm 2 applied 
to R’ converges swiftly and computes the singular values of A with relative accuracy determined 
by |B! 2. 


* Let A have full column rank and let its QR factorization be computed by Businger—Golub column 
pivoting. Write R = ST, where S is the diagonal matrix and T has unit rows in Euclidean norm. 
Then ||T7"[l2 < nl] B' >. 


. [DV05a] The following algorithm carefully combines the properties of the one-sided Jacobi SVD 


and the properties of the QR factorization (Facts 1 and 2): 


Algorithm 3: Preconditioned Jacobi SVD 
(U, 2, [V]) = SVD1(A) 
Input: A € R™*",m>n. 


1: (P, A)P. = Qi 


R ; 
3% R, € R°*", rank revealing QRF. 


2 RT=Q 4 5% Ry € RO*?, 
3: (Uz, Xa, V2) = SVDO( RZ ) 3% one-sided Jacobi SVD on R}. 


U, 0 x) 0 
: — pT 2 : = 2 
Output: U = P7Qi | | Bol - ;; ; 


. [Drm99], [DV05a], [DV05b] 


* Let A and the computed matrix R,; ~ R, in Algorithm 3 be of full column rank. Let Rs be 
computed approximation of R, in Step 2. Then, there exist perturbations 5A, 6R,, and there 
exist orthogonal matrices Q,, Q> such that 


RI 


(1+ 5AAt)A(I + R71SR,) = Qy | | Oy 


Thus, the first two steps of Algorithm 3 preserve the singular values if || B‘||2 is moderate. 


* Let the one-sided Jacobi SVD algorithm with row cyclic pivot strategy be applied to R/. Let the 
stopping criterion be satisfied at the matrix (RI y) during the sth sweep. Write (RI \) = 0%, 
where © is diagonal and U> is numerically orthogonal. Then there exist an orthogonal matrix 
V, and a backward error 5 R, such that 0,5 = (Rp + 5R5)™ Vy, I5(Ra)i:p,i lla < & |(Ra)i-p,illes 
i=1,...,p,withe; < (1+ 6e)°?"-») — 1. If Uy is the closest orthogonal matrix to U2, then 
Ri = YLOT (1 + E), where the dominant part of || E||2 is ||5R2.Ry' ||2. Similar holds for any 
serial or parallel convergent pivot strategy. 


¢ Assembling the above yields 


A 


Vv, 0 


(1+ 5AA')A(I + R7'5Ri) = Qi a 
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The upper bound on the maximal relative error || U~!(X — ¥)||2 is dominated by ||6AA‘||2 + 
RYSRills + IER2Rz Ilo. 
* Let A have the structure A = BD, where full-column rank B has equilibrated columns and D is 


arbitrary diagonal scaling. The only relevant condition number for relative accuracy of Algorithm 
3 is ||B'|[2. 


5. [DV05a], [DV05b] Algorithm 3 outperforms Algorithm 2 both in speed and accuracy. 

6. [Drm99], [Drm00b], [Hig00], [DV05a] It is possible that «2(B) is large, but A is structured as 
A = DCD) with diagonal scalings D,, D2 (diagonal entries in nonincreasing order) and well 
conditioned C. In that case, desirable backward error for the first QR factorization (Step 1) is 
P, D\(C + 6C) D2 P, with satisfactory bound on ||5C||2. This is nearly achieved if the QR factor- 
ization is computed with Powell—Reid’s complete pivoting or its simplification, which replaces row 
pivoting with initial sorting. Although theoretical understanding of this fact is not complete, initial 
row sorting (descending in ||- ||.) norm) is highly recommended. 


Examples: 


1. The key step of the preconditioning is in transposing the computed triangular factors. If A is a 
100 x 100 Hilbert matrix, written as A = BD (cf. Fact 2,), then «,(B) > 10!°. If in Step 2 of 
Algorithm 3, we write Ri as Ri = B,Dj, where D, is diagonal and B, has unit columns in 
Euclidean norm, then k2(B,) < 50. 

2. Take y = 107°, 6 = 107° and consider the matrix 


l yy 1 0 Oo] | 1 y 1} f1 0 0 
A=|-y y y?7}=]0 y oO} J-1 1 1] Jo 1 Of, 
0 6 O 0 0 6}/0 1 Of] Jo 0 y 


with singular values nearly 0; © 1, 0. © y, 03 © 2y6, cf. [DGE99]. A cannot be written as 
A = BD with diagonal D and well-conditioned B. Algorithm 2 computes no accurate digit of 
03, while Algorithm 3 approximates all singular values to nearly full accuracy. (See Fact 5.) The 
computed R7 in Step 2 is 
—1.000000000000000e + 0 0 0 
R= —1.000000000000000e — 20 —1.000000000000000e — 20 0 
—1.000000000000000e — 20 —1.000000000000000e — 40 —2.000000000000000e — 60 


Note that the neither the columns of A nor the columns of R, reveal the singular value of order 10-. 
On the other hand, the Euclidean norms of the column of R/ nicely approximate the singular values 
of A. (See Facts 2, 3.) If the order of the rows of A is changed, R, is computed with zero third row. 


46.3 Accurate SVD from a Rank Revealing Decomposition: 
Structured Matrices 


In some cases, the matrix A is given implicitly as the product of two or three matrices or it can be factored 
into such a product using more general nonorthogonal transformations. For instance, some specially 
structured matrices allow Gaussian eliminations with complete pivoting P} AP, = LDU to compute 
entry-wise accurate [Cx L,D x D,U ~& U. Moreover, it is possible that the triple [,D,U implicitly 
defines the SVD of A to high relative accuracy, and that direct application of any SVD algorithm directly to A 
does not return accurate SVD. For more information on matrices and bipartite graphs, see Chapter 30. For 
more information on sign pattern matrices, see Chapter 33. For more information on totally nonnegative 
(TN) matrices, see Chapter 21. 
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Definitions: 


The singular values of A are said to be perfectly well determined to high relative accuracy if the 
following holds: No matter what the entries of the matrix are, changing an arbitrary nonzero entry 
age to Oaxe, with arbitrary 6 4 0, will cause perturbation 0; ~ 6; € [60;,6,0;], @ = min{|6|, 1/|O]}, 
6, = max{|9|,1/|O|},7 =1,...,n. 

The sparsity pattern Struct(A) of A is defined as the set of indices (k, 2) of the entries of A permitted 
to be nonzero. 

The bipartite graph G(S) of the sparsity pattern S is the graph with vertices partitioned into row 
vertices 7},... ; fm and column vertices c),... ,C,, where rz, and r; are connected if and only if (k,1) € S. 

If G(S) is acyclic, matrices with sparsity pattern S are biacyclic. 

The sign pattern sgn(A) prescribes locations and signs of nonzero entries of A. 

A sign pattern S is total signed compound (TSC) if every square submatrix of every matrix A such that 
sgn(A) = S is either sign nonsingular (nonsingular and determinant expansion is the sum of monomials 
of like sign) or sign singular (determinant expansion degenerates to sum of monomials, which are all 
zero); cf. Chapter 33.2. 

A € R”*" is diagonally scaled totally unimodular (DSTU) if there exist diagonal D,, D, and totally 
unimodular Z (all minors of Z are from {—1, 0, 1}) such that A = D, ZD). 

A decomposition A = X DY" with diagonal matrix D is called an RRD if X and Y are full-column 
rank, well-conditioned matrices. (RRD is an abbreviation for “rank revealing decomposition,” but that 
term is defined slightly differently in Chapter 39.9, where X,Y are required to be orthogonal and D is 
replaced by an upper triangular matrix.) 


Facts: 


1. [DG93] The singular values of A are perfectly well determined to high relative accuracy if and only 
if the bipartite graph G(S) is acyclic (forest of trees). Sparsity pattern S with acyclic G(S) allows at 
most m+ n — 1 nonzero entries. A bisection algorithm computes all singular values of biacyclic 
matrices to high relative accuracy. 

2. [DK90], [FP94] Bidiagonal matrices are a special case of acyclic sparsity pattern. Let n x n bidiagonal 
matrix B be perturbed by bj; ~ 0;{&2;-1, bij41 ~> bij41€2; for all admissible is, and let ¢ = 
We: max(|g;|, 1/|€;|), where all ¢; 4 0. If) > --- > &, are the singular values of the perturbed 
matrix, then for all i, o;/¢ < 6; < €0;. The singular values of bidiagonal matrices are efficiently 
computed to high relative accuracy by the zero shift QR algorithm and the differential qd algorithm. 

3. [DGE99] Let S, be a sparsity-and-sign pattern. Let each matrix A with pattern S, have the 
property that small relative changes of its (nonzero) entries cause only small relative perturbations 
of its singular values. This is equivalent with S being total signed compound (TSC). 

4, [Drm98a], [DGE99] Let A be given by an RRD A = XD Y". Without loss of generality assume 
that X and Y have equilibrated (e.g., unit in Euclidean norm) columns. Algorithm 4 computes the 
SVD of A to high relative accuracy. 


Algorithm 4: RRD Jacobi SVD 
(U, Xx, V) = SVD2(X, D, Y) 
Input: X € R”*?, Y € R"*? full column rank, D € R?*? diagonal. 


1: Y=YD;YP=Q % rank revealing QR factorization. 


23 Z =(XP)R? ; % explicit standard matrix multiplication. 
3: (Uz, Xz, Vz) = SVD1(Z) % one-sided Jacobi SVD on Z. 
Output: U = U,; 2 = 2s Oe Vac 28 .XDYT =UXV'!. 
0 0 OP Tis 
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5. 


6. 


[Drm98a], [DGE99], [DM04] 


* In Step 2 of Algorithm 4, the matrix RT has the structure R? = LS, where S is diagonal scaling, 
L has unit columns, and || ~' || is bounded by a function of p independent of input data. The 
upper bound on «3(L) depends on pivoting P in Step 1 and can be O(n!+(!/4)l0&”). For the 
Businger—Golub, pivoting the upper bound is O(2"), but in practice one can expect an O(n) 
bound. 


* The product Z = (XP)R‘ must not be computed by fast (e.g., Strassen) algorithm if all singular 
values are wanted to high relative accuracy. The reason is that fast matrix multiplication algorithms 
can produce larger component-wise perturbations than the standard O(n3) algorithm. 


* The computed U,z,V satisfy: There exist orthogonal UU, orthogonal V x V,and E 1, E2 
such that USV7 = (I + E,)A(I + Ea), || Eill2 < O(€)K2(X), || Eall2 < O(€)k2(L)k2(Y) . The 
relative errors in the computed singular values are bounded by O(€)k2(L) max{k2(X), k2(Y)}. 
Here, O(€) denotes machine precision € multiplied by a moderate polynomial in m, n, p. 


[DGE99] Classes of matrices with accurate LDU factorization and, thus, accurate SVD computa- 
tion by Algorithm 4, or other algorithms tailored for special classes of matrices (e.g., QR and qd 
algorithms for bidiagonal matrices), include: 


* Acyclic matrices: Accurate LDU factorization with pivoting uses the correspondence between the 
monomials in determinant expansion and perfect matchings in G(S). 


Total signed compound (TSC) matrices: LDU factorization with complete pivoting P, AP, = 
LDU ofan TSC matrix A can be computed in a forward stable way. Cancellations in Gaussian 
eliminations can be avoided by computing some elements as quotients of minors. All entries of 
L, D, U are computed to high relative accuracy. The behavior of «(L) and «(U) is not fully 
analyzed, but they behave well in practice. 


Diagonally scaled totally unimodular (DSTU) matrices: This class includes acyclic and certain 
finite element (e.g., mass-spring systems) matrices. Gaussian elimination with complete pivoting 
is modified by replacing dangerous cancellations by exactly predicted exact zeros. All entries of 
L, D, U are computed to high relative accuracy, and «(L), «(U) are at most O(mn) and O(n’), 
respectively. 


[Dem99] In some cases, new classes of matrices and accurate SVD computation are derived from 
relations with previously solved problems or by suitable matrix representations. The following cases 
are analyzed and solved with O(n*) complexity: 


* Cauchy matrices: C = C(x, y), ci; = 1/(x; + y;), and the parameters x;s, y;s are the initial data. 
The algorithm extends to Cauchy-like matrices C’ = D,C D2, where C is a Cauchy matrix and 
Dj, D2 are diagonal scalings. The required LDU factorization is computed with én? operations 
and given as input to Algorithm 4. It should be noted that this does not imply that Cauchy 
matrices determine their singular values perfectly well. The statement is: For a Cauchy matrix C, 
given by set of parameters x;’s, y;’s stored as floating point numbers in computer memory, all 
singular values of C can be computed in a forward stable way. Thus, for example, singular values 
of a notoriously ill-conditioned Hilbert matrix can be computed to high relative accuracy. 


* Vandermonde matrices: V = [vij], vij = x? ~!. An accurate algorithm exploits the fact that 
postmultiplication of V by the discrete Fourier transform matrix gives a Cauchy-like matrix. The 
finite precision arithmetic requires a guard digit and extra precision to tabulate certain constants. 


* Unit displacement rank matrix X: Solution of matrix equation AX + XB = dd} , where A, 
B are diagonal, or normal matrices with known accurate spectral decompositions. The class of 
unit displacement rank matrices generalizes the Cauchy-like matrices (A and B diagonal) and 
Vandermonde matrices (A diagonal and B circular shift). 
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8. [DK04] Weakly diagonally dominant M-matrices given with a;; < 0 fori ¢ j and with the row- 
sums s; = eee aj; = 0,1 = 1,..., known with small relative errors, allow accurate LDU 
factorization with complete pivoting. 

9. [Koe05] Totally nonnegative (TN) matrix A can be expressed as the product of nonnegative bidiag- 
onal matrices. If A is given implicitly by this bidiagonal representation, then all its singular values 
(and eigenvalues, too) can be computed from the bidiagonals to high relative accuracy. Accurate 
bidiagonal representation for given TN matrix A is possible for totally positive cases (all minors 
positive), provided certain pivotal minors can be computed accurately. 

10. [DK05a] The singular values of the matrix V with entries vj; = P;(x;), where the Pjs are ortho- 
normal polynomials and the x;s are the nodes, can be computed to high relative accuracy. 

11. [Drm00a] Accurate SVD of the RRD X DY" extends to the triple product XSY", where S is not 
necessarily diagonal, but it can be accurately factored by Gaussian eliminations with pivoting. 


Examples: 


1. An illustration of the power of the algorithms described in this section is the example of a 
100 x 100 Hilbert matrix described in [Dem99]. Its singular values range over 150 orders of mag- 
nitude and are computed using the package Mathematica with 200-decimal digit software floating 
point arithmetic. The computed singular values are rounded to 16 digits and used as reference 
values. The singular values computed in IEEE double precision floating point (€ © 107!°) by the 
algorithms described in this section agree with the reference values with relative error less than 
34 -€. 

2. [DGE99] Examples of sign patterns of TSC matrices: 


+ 0 0 0 + ++ + + 
ae ee ED? 1D fe. Ne. Qe) S203 70 
0 Oi iota Oe =. 108 0 
OF. eee AB Ee S02 Ol =e 20 
0 0 0 Os 30? Or 2S 


TSC matrices must be sparse because an m x n TSC matrix can have at most 3 (m+n) —2 nonzero 
entries. 


46.4 Positive Definite Matrices 


Definitions: 
The Cholesky factorization with pivoting of symmetric positive definite n x n matrix H computes the 
Cholesky factorization PH P = LL", where the permutation matrix P is such that £7, > Sj_; ee BB 
l<i<j<n. 
The component-wise relative distance between H and its component-wise relative perturbation H is 
reldist(H, H) = max aa where 0/0 = 0. 
ij ij 
A diagonally scaled representation of symmetric matrix H with positive diagonal entries is a factored 
hj; 
representation H = DAD with D = diag(./hy),... »W Ann), aij = T= 
i ij 
Facts: 
1. [Dem89], [Dem92] Let H = DAD be a diagonally scaled representation of positive definite H, 
and let Amin( A) denote the minimal eigenvalue of A. 
° If 5H is a symmetric perturbation such that H + 6H is not positive definite, then 
|ohi;\ > Amin(A) 1 


max = ; 
Isijsn \/hijhjj ~ on n|| A |l2 
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5h;; 
° If SH = —Amin(A)D?, then aay = Amin(A), reldist(H, H + 6H) = Apin(A), and 


oJ a/ hi; h ii 
H + 6H is singular. 
2. [Dem89] Let H = DAD be ann x n symmetric matrix with positive diagonal entries, stored in 
the machine memory. Let H be the input matrix in the Cholesky factorization algorithm. Then the 
following holds: 


* If the Cholesky algorithm successfully completes all operations and computes lower triangular 
matrix [, then there exists symmetric backward perturbation 6H such that [l?’ =H+6H 


and |d5hij| < ney/hihjj,nc < O(ne). 


* IfAmin(A) > nyc, then the Cholesky algorithm will succeed and compute Ls 

° IfAmin(A) < e, then there exists simulation of rounding errors in which the Cholesky algorithm 
fails to complete all operations. 

° IfAmin(A) < —nnc, then it is certain that the Cholesky algorithm will fail. 

3. [DV92] If H = DAD € PD, is perturbed to a positive definite H = H + 6H, then 
him hi D'6HD"! 

mig Si ee 
lsi<n Xj Amin (A) 

* Let 5H = nD’, with any 7 € (0,Amin(A)). Then for some index i it holds that 


[Ai — Ail 7 
—— = 1+ : 
Xi Amin(A) 


4. [DV92], [VS93] Let H = DAD be positive definite and c > 0 constant such that for all ¢ € 
(0, 1/c) and for all symmetric component-wise relative perturbations 5H with |6h;;| < e|hijl, 
4 di — Ai 
1 <i,j <n, the ordered eigenvalues 4; and A; of H and H + 6H satisfy max mel <c 
Then ||A7'||2 < (1 +c)/2. The same holds for more general perturbations, e.g., 5H with \ohij| < 
Ex/ hi; hij . 


5. [vdS69] It holds that || A7'|]2 < «2(A) < n min k2o(DHD) < nk2(H). 
=diag 


= |A"|2||D-'SHD™ I>. 


&s 


6. For a general dense positive definite H = DAD stored in the machine memory, eigenvalue com- 
putation with high relative accuracy is numerically feasible if and only if Amin(A) is not smaller 
than the machine round-off unit. It is possible that matrix H is theoretically positive definite and 
that errors in computing its entries as functions of some parameters cause the stored matrix to be 
indefinite. Failure of the Cholesky algorithm is a warning that the matrix is entry-wise close to a 
symmetric matrix that is not positive definite. 

7. [VH89] If P’ HP = LL? is the Cholesky factorization with pivoting of positive definite H, then 
the SVD L = UXV’" of L is computed very efficiently by the one-sided Jacobi SVD algorithm, 
and H is diagonalized as H = (PU)=?(PU)!. 


Algorithm 5: Positive definite Jacobi EVD 
(A, U) = EIG,(H) 
Input: H € PD, ; 
P?HP =LL! ; % Cholesky factorization with pivoting. 
if L computed successfully, then 
(U, X) = SVDO(L) % One-sided Jacobi SVD on L. V is not computed. 
AG = U2, Utng = PUinj, i=1,...513 
Output: A = (Aq,...5An)3U 
else 
Error: H is not numerically positive definite 
end if 
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8. [DV92], [Mat96], [Drm98b]. Let 4; > --- > A, be the approximations of the eigenvalues A, > 
+++ >A, of H = DAD, computed by Algorithm 5. Then 


* The computed approximate eigenvalues of H are the exact eigenvalues of a nearby symmetric 
ee |Shijl 
ositive definite matrix H + 6H, where max ——— < ¢, and ¢ is bounded by O(n) times 
P 1S<i,j<n Shih jj y, 
the round-off unit e. 
Ai — Ail 
e max eats _. = a 


< né||A7'||2. The dominant part in the forward relative error is committed 
l<i<n 


1 


during the Cholesky factorization. The one-sided Jacobi SVD contributes to this error with, at 
most, O(n)ev/|| Aq! |]2 + O(n’ )e. 
9. Numerical properties of Algorithm 5, given in Fact 8, are better appreciated if compared with 


algorithms that first reduce H to tridiagonal matrix T and then diagonalize T. For such triadiago- 
nalization based procedures the following hold: 


* The computed approximate eigenvalues of H are the exact eigenvalues of a nearby symmetric 
matrix H + 6H, where ||6 H||2 < ¢|| H||2 and ¢ is bounded by a low degree polynomial in n times 
the round-off unit e. 

* The computed eigenvalue approximations Ai dj satisfy the absolute error bound |A; — hil < 


&|| Hz, that is, max ccd < €k2(H). 

10. In some applications it might be possible to work with a positive definite matrix implicitly as 
H = C'C, where only a full column rank C is explicitly formed. Then the spectral computation 
with H is replaced with the SVD of C. The Cholesky factor L of H is computed implicitly from 
the QR factorization of C. This implicit formulation has many numerical advantages and it should 
be the preferred way of computation with positive definite matrices. An example is natural factor 
formulation of stiffness matrices in finite element computations. 

11. [Drm98a], [Drm98b] Generalized eigenvalues of HM — AI and H — 1M can be computed to 
high relative accuracy if H = Dy AyDy, M = Dy Am Dy with diagonal Dy, Dy and moderate 
[Age lias App l2- 


Examples: 


1. In this numerical experiment we use MATLAB 6.5, Release 13 (on a Pentium 4 machine under MS 
Windows® 2000), and the function eig(-) for eigenvalue computation. Let 


104° 1029 1019 
H= 1|102? 107° 10° 
10° 10° 1 


The sensitivity of the eigenvalues of H and the accuracy of numerical algorithms can be illustrated 
by applications of the algorithms to various functions and perturbations of H. Let P’ HP be 
obtained from H by permutation similarity with permutation matrix P. Let H + AH be obtained 
from H by changing Hz) into — Hy. = —10”°, and let H + 5H be obtained from H by multiplying 
Hy3 and H3, by 1 + €, where € © 2.22 - 107'° is the round-off unit in MATLAB. For the sake 
of experiment, the eigenvalues of numerically computed (H~')~! are also examined. The values 
returned by eig() are shown in Table 46.1. All six approximations of the spectrum of H are with 
small absolute error, max al “ AL | < O(e). Some results might be different if a different version 
of MATLAB or operating system is used. 

2. Let H be the matrix from Example 1. H is positive definite, k.(H) > 10°, and its H = DAD 
representation with D = diag(107°, 10°, 1) gives «,(A) < 1.4, || A7!|l2 < 1.2. The Cholesky factor 
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TABLE 46.1 
eig(H) eig(P’HP), P ~ (2,1,3) eig(inv(inv(H))) 
dy 1.000000000000000e + 040 1.000000000000000e + 040 1.000000000000000e + 040 


Az —8.100009764062724e +019 9.900000000000000e + 019 = 9.900000000000000e + 019 
is  —3.966787845610502e + 023 —9.818181818181818e —001 9.818181818181817e — 001 


eig(H + AH) eig(P’ HP), P ~ (3,2,1) eig(H+65H) 
1 1.000000000000000e + 040 1.000000000000000e + 040 1.000000000000000e + 040 


r 
hy  —8.100009764062724e + 019 — 9.900000000000000e + 019 —_-1.208844819952007e + 024 
is  —3.966787845610502e + 023 9.818181818181819e — 001 —9.899993299416013e — 001 


L of H is successfully computed in MATLAB by the chol(-) function. The matrix L7L, which is 
implicitly diagonalized in Algorithm 5, reads 


1.000000000e + 40 9.94987437le +18 9.908673886e — 02 
H =L'L = |9.94987437le +18 9.900000000e + 19 8.962732759e — 02 
9.908673886e —02 8.962732759e —02 9.818181818e — 01 


The diagonal of H approximates the eigenvalues of H with all shown digits correct. To see that, 
write H as H = DAD, where D = diag(V/ An, V Hy, Vv Hy3) and 
1.000000000e + 00 1.000000000e — 11 1.000000000e — 21 
A = |1.000000000e — 11 1.000000000e +00 9.090909091e — 12 
1.000000000e — 21 9.090909091le — 12 1.000000000e + 00 


with ||A — || < 1.4-107!!, ||A7||, © 1. Algorithm 5 computes the eigenvalues of H, as 
Ay © 1.0e + 40, A2 © 9.900000000000002e + 19,13 + 9.818181818181817e — 01. 

3. Smallest eigenvalues can be irreparably damaged simply by computing and storing matrix entries. 
This rather convincing example is discussed in [DGE99]. The stiffness matrix of a mass spring 
system with 3 masses, 


kth —-k 0 
K= | —-ky kh+kj; —k3|, ky, ko, ks spring constants, 
0 —k; ks 


is computed with k; = k3 = 1 and ky = €/2, where € is the round-off unit. Then the true and the 
computed assembled matrix are, respectively, 


l+e/2 -e/2 0 1 -e/2 0 
K=]| -e/2 1+e€/2 -1], K=]-e/2 1 -1 
0 =A 1 0 =) Ce 


K is the component-wise relative perturbation of K with reldist(K, K) =e/(2+€) <€/2.K is 
positive definite with minimal eigenvalue near €/4, K is indefinite with minimal eigenvalue near 
—e?/8. MATLAB’s function chol(-) fails to compute the Cholesky factorization of K and reports 
that the matrix is not positive definite. 

On the other hand, writing K = A’ Awith 


ky 0 0 Sky 0 0 1 0 0 
A=|-J/k Jk O|]=] 0 Vb oO; |-1 1 °«0 
0  —VSks ks OO nafs} | 0 1 A 
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clearly separates physical parameters and the geometry of the connections. Since A is bidiagonal, 
for any choice of ky, k, k3, the eigenvalues of K can be computed as squared singular values of A 
to nearly the same number of accurate digits to which the spring constants are given. 


46.5 Accurate Eigenvalues of Symmetric Indefinite Matrices 


Relevant relative perturbation theory for floating point computation with symmetric indefinite matrices 
is presented in [BD90], [DV92], [VS93], [DMMO0], and [Ves00]. Full review of perturbation theory is 
given in Chapter 15. 


Definitions: 


Bunch-Parlett factorization of symmetric H is the factorization P THP = LBL’, where P isa permu- 
tation matrix, B is a block-diagonal matrix with diagonal blocks of size 1 « 1 or 2 x 2, and L is a full 
column rank unit lower triangular matrix, where the diagonal blocks in L that correspond to 2 x 2 blocks 
in B are 2 x 2 identity matrices. 

A symmetric rank revealing decomposition (SRRD) of H is a decomposition H = X DX7, where D 
is diagonal and X is a full column rank well-conditioned matrix. 

Let 7 denote a nonsingular symmetric matrix. Matrix B is 7-orthogonal if F TIF=aZJ. (Warning: 
this is a nonstandard usage of F, since in this book F usually denotes a field.) 

The hyperbolic SVD decomposition of the matrix pair (G, 7) isa decomposition of G, G = WX FI, 
where W is orthogonal, & is diagonal, and F is 7-orthogonal. 


Facts: 


1. If H = UXV’" is the SVD of an n x n symmetric H, where © = @'",0;,1,,, then the matrix 
V'Uis block-diagonal with m symmetric and orthogonal blocks of sizes nj x nj,i = 1,...,m 
along its diagonal. The n; eigenvalues of the ith block are from {—1, 1} and they give the signs of 
n; eigenvalues of H with absolute value ,. 

2. If H = GJG' isa factorization with full column rank G € R"*’ and J = diag(+1), then the 
eigenvalue problems Hx = Ax and(G?G)y = AJZy are equivalent. If F is the 7 orthogonal eigen- 
vector matrix of the pencil G'G —AJ (F'IJF = J, F'(G'G)F = &* = diag(o?,... ,07)), 
then the matrix £?.7 = diag(A,,... ,A,) contains the nonzero eigenvalues of H with the columns 
of (GF)X~! as corresponding eigenvectors. 

3. [Ves00] Let H have factorization H = G.JG! as in Fact 2. Suppose that H is perturbed implicitly 
by changing G ~» G+6G, thus H =(G+6G)7(G+6G)". Write G = BD, where D is diagonal 
and B has unit columns in the Euclidean norm, and let 6B = 5GD~!. Let 9 = ||6BB'||, < 1. 


|dAj 


4, [Sla98], [Sla02] The factorization H = GJG? in Fact 2 is computed by a modification of Bunch— 
Parlett factorization. Let G be the computed factor andlet 7 = diag(+1) be the computed signature 
matrix. Then 
* A backward stability relation H + 5H = GIG" holds with the entry-wise bound |6H| < 

O(n)e(|H| + |GI|GI"). 
* Let G have rank n. If J = J andifa, > --- > A, are the exact eigenvalues of the pencil 
O(n*)e 

= G2, (D-IGF) 

such that D~!G has unit rows in Euclidean norm, F is the eigenvector matrix of GIG =i: 

and Omin(-) denotes the minimal singular value of a matrix. 


A 


G'G — AZ, then, for all i, |A; — Aj| |A;|, where D denotes a diagonal matrix 
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5. [Ves93] The one-sided (implicit) 7-symmetric Jacobi algorithm essentially computes the hyper- 
bolic SVD of (G, 7), (W, x, F) = HSVD(G, 7). It follows the structure of the one-sided Jacobi 
SVD (Algorithm 2) with the following modifications: 


* On input, A is replaced with the pair (G, 7). 


* In step k, G4t) = GY is computed from G“) using Jacobi plane rotations exactly as in 
Algorithm 2 if J; and Jj; are of the same sign. (Here, i = ix, j = jx are the pivot indices in 
G®.) 


If J;; and 7J;; have opposite signs, then the Jacobi rotation is replaced with hyperbolic rotation 


2& 
d; +d;° 


i & sinh 


: , tanh2q, = — 
sinh € cosh fq 


gé= (G)F (G )injo d= (G)JT (G™)ines £ = i,j. The tangent is determined as 
sign(tanh 2¢;) 


| tanh 2¢;| + 4/ tanh? 2¢;, — 1 


The limit of G™ is WX, and the accumulated product VOY ...yH... converges to J- 
orthogonal F, and G = WX F~! is the hyperbolic SVD of G. 


‘ A é ; (CG) 7 (GE) inj ‘ 
The iterations are stopped at index k if max ; = . < t. The tolerance Tt is 
ifj |(G! uni llell CG inj lle 


tanh &, = 


usually set to ne. 


6. [Ves93] The eigenvalue problem of a symmetric indefinite matrix can be implicitly solved as a 
hyperbolic SVD problem. Algorithm 6 uses the factorization H = GIG" (Fact 2) and hyperbolic 
SVD HSVD(G, 7) (Fact 5) to compute the eigenvalues and eigenvectors of H. 


Algorithm 6: Hyperbolic Jacobi 
(A, U) = EIGO(H) 
Input: H € S, 
1: H=GIG' ,J= Ip ® Un—p) 3 % Bunch—Parlett factorization (modified). 
2: (W, X, F) = HSVD(G, 7) % One-sided J-symmetric Jacobi algorithm. 
3: Mi = Fi U2 3 Ving = Wins i = 1,...573 
Output: A = (Ay,...5An)3U 


7. [Sla02] Let GO = G, G™,k = 1,2,... be the sequence of matrices computed by the one-sided 
J -symmetric Jacobi algorithm in floating point arithmetic. Write each G™ asG™ = BY p&, 
where D“? is the diagonal matrix with Euclidean column norms of G“ along its diagonal. 


* G+) js the result of an exactly 7-orthogonal plane transformation of G® + 6G“ = (BW + 
5B) D®, where ||SB“ ||» < c,e with moderate factor cx. 
* Let Ay > --- > A, be computed as Fi(G)E, (Gnas i = 1,...,n, where G™) is the 
first matrix which satisfies stopping criterion. If k’ is the index of last applied rotation, then 
hij = hi A z - 
max ome < 2n+n’, with); asin Fact 4,andy = O(e)S> (B®), + O(n)e + O(e?). 


Isisn | Aj| ja0 


8. [DMM03] Accurate diagonalization of indefinite matrices can be derived from their accurate SVD 
decomposition. (See Fact 1.) 
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Algorithm 7: Signed SVD 
(A, Q) = EIGI(H) 
Input: H € S, 
1 H = XDY! ; % rank revealing decomposition. 
2 (U, x, V) = SVD2(X, D, Y) ; % accurate SVD. 
3: Recover the signs of the eigenvalues, 4; = -o;, using the structure of viuU; 
4 Recover the eigenvector matrix Q using the structure of V7U. 
Output: A = (Aj,...5A4n) 3 Q 


Let H © USV", ¥ = diag(&1,... ,,), be the SVD computed by Algorithm 7. 


* If all computed singular values ¢;,i = 1,... ,n, are well separated, Algorithm 7 chooses i= 
Gsign(V in, Tirni)s with the eigenvector Quni — Vins 
* Ifsingular values are clustered, then clusters are determined by perturbation theory and the signs 


are determined inside each individual cluster. 


[DMM03] If the rank revealing factorization in Step 1 of Algorithm 7 is computed as H * X DY? 
with ||X — X|| < &, ||Y — Y|| < & maxjey., [dij — dii|/|dii| < &, then the computed eigenvalues 
Aj satisfy |A; — A;| < &’«(L) max{«(X),«(Y)}|A;|, i = 1,...,n. Here, € and &’ are bounded by 
the round-off € times moderate functions of the dimensions, and it is assumed that the traces of 
the diagonal blocks of V"U (Cf. Fact 1) are computed correctly. 

[DK05b] An accurate symmetric rank revealing decomposition H = XDXT™ can be given as 
input to Algorithm 7 or it can replace Step 1 in Algorithm 6 by defining G = X./|D], J = 
diag(sign(d),),... , sign(dyy)). Once an accurate SRRD is available, the eigenvalues are computed to 
high relative accuracy. For the following structured matrices, specially tailored algorithms compute 
accurate SRRDs: 


* Symmetric scaled Cauchy matrices, H = DCD, D diagonal, C Cauchy matrix. 
* Symmetric Vandermonde matrices, V = Cea imas omen where v ER. 


° Symmetric totally nonnegative matrices. 


Examples: 


ie 


[Ves96] Initial factorization or rank revealing decomposition is the key for success or failure of the 
algorithms presented in this section. Let ¢ = 107” and 


L-acot 10 0|]fo1 ol] fi 0 o 
H=|1 0 oJ}J=]0 1 offi 0 of] joi 1 
106 0 1 Vel lo 0 1] ]o 0 ve 


This factorization implies that all eigenvalues of H are determined to high relative accuracy. Whether 
or not they will be determined to that accuracy by Algorithm 6 (or any other algorithm depending 
on initial factorization) depends on the factorization. In Algorithm 6, the factorization in Step 1 
chooses to start with 1 x 1 pivot, which after the first step gives 


1 0 ° 1 0 0 oa ] 

1 1 0; ;O -1 —1 0 1 Oj. 
0 0 | 

The 2 x 2 Schur complement is ill-conditioned (entry-wise ¢ close to singularity, the condition 

number behaving as 1/¢) and the smallest eigenvalue is lost. 


1 0 | 0 —l —l+e 
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Multiplying matrices is an important problem from both the theoretical and the practical point of view. 
Determining the arithmetic complexity of this problem, that is, the minimum number of arithmetic 
operations sufficient for computing an n x n matrix product, is still an open issue. Other important 
computational problems like computing the inverse of a nonsingular matrix, solving a linear system, 
computing the determinant, or more generally the coefficients of the characteristic polynomial of a matrix 
have a complexity related to that of matrix multiplication. Certain combinatorial problems, like the all 
pair shortest distance problem ofa digraph, are strictly related to matrix multiplication. This chapter deals 
with fast algorithms for multiplication of unstructured matrices. Fast algorithms for structured matrix 
computations are presented in Chapter 48. 


47.1 Basic Concepts 


Let A = [a;,j], B = [b;,;], and C = [c;,;] be n x n matrices over the field F such that C = AB, that is, 
C is the matrix product of A and B. 


Facts: 


1. The elements of the matrices A, B, and C are related by the following equations: 


n 
Ci,j = abies 1, J Silyens 
k=1 
2. Each element c;,; is the scalar product of the ith row of A and the jth column of B and can be 
computed by means of n multiplications and n — 1 additions. This computation is described in 
Algorithm 1. 
3. The overall cost of computing the n? elements of C is n> multiplications and n> — n? additions, 
that is 2n? — n? arithmetic operations. 
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Algorithm 1. Conventional matrix multiplication 


Input: the elements a;,; and bj,; of two n x n matrices A and B; 
Output: the elements c;,; of the matrix product C = AB; 
fori = 1 tondo 
for j = 1tondo 
Ci,j =0 
fork = 1tondo 
Cig = Ci,j + Gikdk,j 


4. [Hig02, p.71] The computation ofc;,; bymeans of Algorithm 1 is element-wise forward numerically 
stable. More precisely, if C denotes the matrix actually computed by performing Algorithm 1 in 
floating point arithmetic with machine precision €, then |C — C| < ne|A||B| + O(e7), where | A| 
denotes the matrix with elements |a;,;| and the inequality holds element-wise. 


Examples: 


1. For 2 x 2 matrices Algorithm 1 requires 8 multiplications and 4 additions. For 5 x 5 matrices 
Algorithm 1 requires 125 multiplications and 100 additions. 


47.2 Fast Algorithms 


Definitions: 


Define w € R as the infimum of the real numbers w such that there exists an algorithm for multiplying 
n X n matrices with O(n") arithmetic operations. @ is called the exponent of matrix multiplication 
complexity. 
Algorithms that do not use commutativity of multiplication, are called noncommutative algorithms. 
Algorithms for multiplying the p x p matrices A and B of the form 


P Pp 
a Py i,j,k Gi, j Ss Bijebij ), k=1,...t, 


i,j=l ij=l 


t 
Cj = S Yi, j.kIMK» 
k=1 


where aj, ;,4, Bi, j,k Vi,j,k ave given scalar constants, are called bilinear noncommutative algorithms with t 
nonscalar multiplications. 


Facts: 


1. [Win69] Winograd’s commutative algorithm. For moderately large values of n, it is possible to 
compute the product of n x n matrices with less than 2n? — n? arithmetic operations by means of 
the following simple identities where n = 2m is even: 


m m 
uj = S Gi2k-14i,2k) Vi = S box-1,jb2,j, 1 = 1,n, 
k=l 


k=l 
m 
Wig = s (di,2e—1 + box, ;)(@i2e + b2-1,)), bj = 140, 
k=l 
Ci,j = Wij — Ui — Dj, i,j =1,n. 


2. The number of arithmetic operations required is n> + 4n? — 2n, which is less than 2n? — n? already 
for n > 8. This formula, which for large values of n is faster by a factor of about 2 with respect to the 
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11. 


conventional algorithm, relies on the commutative property of multiplication. It can be extended 
to the case where n is odd. 


. [Str69] Strassen’s formula. It is possible to multiply 2 x 2 matrices with only 7 multiplications 


instead of 8, but with a higher number of additions by means of the following identities: 


m, = (41,2 — 42,2)(b2,1 + b22), ms = a1,(b1,2 — b22), 
Mz = (41,1 + 42,2)(b1,1 + b22), Mp = 42,2(b2,1 — bi), 
m3 = (a,1 — 42,1)(b1,1 + b1,2), m7 = (a2, + 422)bi 1, 


mg = (4,1 + 41,2) b2,, 
C1 = My +m,-— M+ Me, C12 = M+ Ms, 
C21 = Me + M7, €2,2 = Mz — m3 + M5 — M7. 


. The overall number of arithmetic operations required by the Strassen formula is higher than the 


number of arithmetic operations required by the conventional matrix multiplication described in 
Algorithm 1 of section 47.1. However, the decrease from 8 to 7 of the number of multiplications 
provides important consequences. 


. The identities of Fact 3 do not exploit the commutative property of multiplication like the identities 


of Fact 1, therefore they are still valid if the scalar factors are replaced by matrices. 


. Strassen’s formula provides a bilinear noncommutative algorithm where the constants qj, ;,x, Bj, j,k, 


Vi,j,k are in the set {0, 1, —1}. 
If n is even, say n = 2m, then A, B, and C can be partitioned into four m x m blocks, that is, 


op By Biz = Cir Cia 

Bz, B22 Cr1 Cr 
where Aj, ;, Bj,j,Cj,; € F”*™, i, j = 1,2, so that ann x n matrix product can be viewed as a2 x 2 
block matrix product. More specifically it holds that 


Cy = Ai Bi + Ai2Ba1,  Ci2 = Ar1Bi2 + Ai,2B2,; 
Cy1 = Az Bi + A22B21, C22 = A21 Biz + A2,2 B22. 


If n = 2m, the four blocks C;,; of Fact 7 can be computed by means of Strassen’s formula with 
7m X m matrix multiplications and 18 m x m matrix additions, i.e., with 7(2m> — m*) + 18m? 
arithmetic operations. The arithmetic cost of matrix multiplication is reduced roughly by a factor 
of 7/8. 

[Str69] Strassen’s algorithm. Furthermore, if m is even then the seven m x m matrix products of 
Fact 8 can be computed once again by means of Strassen’s formula. If n = 2*, k a positive integer, 
Strassen’s formula can be repeated recursively until the size of the blocks is 1. Algorithm 2 synthesizes 
this computation. 

The number M(k) of arithmetic operations required by Algorithm 2 to multiply 2‘ x 2* matrices 
is such that 


M(k) = 7M(k — 1) +18 (2*-1)’, 


which provides M(k) = 7-7* — 6. 4* = 7n'°&7 — 6n?, where n = 2* and log, 7 = 2.8073... < 3. 
This yields the bound w < log, 7 on the exponent w of matrix multiplication complexity. 

In practice it is not convenient to carry out Strassen’s algorithm up to matrices of size 1. In 
fact, for 2 x 2 matrices Strassen’s formula requires much more operations than the conventional 
multiplication formula (see Example 1). Therefore, in the actual implementation the recursive 
iteration of Strassen’s algorithm is stopped at size p = 2", where p is the largest value such that the 
conventional method applied to p x p matrices is faster than Strassen’s method. 
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12. Strassen’s algorithm can be carried out even though n is not an integer power of 2. Assume that 
1 <n < 2, set p= 2* — n, and embed the matrices A, B,C into matrices A, B, C of size 2‘ 


in the following way: 
A 0, “a BO, A C0, 
" ? 7 "| = | "| 
Opn Opp Opn Opp Opn Opp 


Then one has C = AB so that Strassen’s algorithm can be applied to A and B in order to compute 
C. Even in this case the cost of Strassen’s algorithm is still O( 1'°827), 


_~ 


Algorithm 2. Strassen’s algorithm 


Procedure Strassen(k, A,B) 
Input: the elements a;,;and b;,; of the 2* x 2* matrices A and B; 
Output: the elements c;,; of the 2k x 2* matrix C = AB; 


If k = 0 then 
output c11 — 41,101,13 
else 
partition A, B, and C into 2‘~! x 2—! blocks Aj,;, Bi,j, Ci,j, respectively, 
where i, j = 1,2; 
compute 
P, = Aj — Az2, Qi = Ba + Bay, Ps = Aji, Qs = Biz — B22, 
P2 = Aj + Az2, Q2 = Bi + Bayz, Pe = A222, Q6 = B21 — Bi, 
P3 = Aj — Agi, Q3 = Bi + Bi, P7 = Az + Az2, Q7 = Bia, 


Py = Aji + Aiz, Qa = B22, 
fori = 1to7do 
M; = Strassen (k — 1, P;, Q;); 


compute 
Cyr = M+ M.—-My+ Me, Ci2 = My + Ms, 
C21 = Me + M7, C22 = Mz — M3 + Ms — M7. 


13. [Hig02, p. 440] Numerical stability of Strassen’s algorithm. Let C be the n x n matrix obtained 
by performing Strassen’s algorithm in floating point arithmetic with machine precision € where 
n = 2", Then the following bound holds: max;, ; [¢i,; —ci,j| < Yn€ maxi, |aj,;| max;,; |i,;|+ O(e?), 
where y, = 6n'°&? and log, 12 ~ 3.585. Thus, Strassen’s algorithm has slightly less favorable 
stability properties than the conventional algorithm: the error bound does not hold component- 
wise but only norm-wise, and the multiplicative factor 6n'°% !” is larger than the factor n? of the 
bound given in Fact 4 of section 47.1. 


Examples: 


1. For n = 16, applying the basic Strassen algorithm to the 2 x 2 block matrices with blocks of size 
8 and computing the seven products with the conventional algorithm requires 7 * (2 * 8° — 8°) + 
18 « 8? = 7872 arithmetic operations. Using the conventional algorithm requires 2 * 16° — 16? = 
7936 arithmetic operations. Thus, it is convenient to stop the recursion of Algorithm 2 when 
n = 16. A similar analysis can be performed if the Winograd commutative formula of Fact 1 
is used. 
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Other Algorithms 


Facts: 
iB 


10. 


[Win71] Winograd’s formula. The following identities enable one to compute the product of 2 x 2 
matrices with 7 multiplications and with 15 additions; this is the minimum number of additions 
among bilinear noncommutative algorithms for multiplying 2 x 2 matrices with 7 multiplications: 
S$] = 421 1422, S2=S$1—- 411, $3 = 411 — 421, $4 = 412 — S25 
ha=by-b, h=brz-h, b=hp-bira, t~=h— ba, 


mM, = 4,10), mz = 41,262, m3 = s4b22, m4 = ag214, 


Ms = Sh, Me = S2br, M7 = $313, 
uy =mM +m, U=uUj+m, UW=u;+mMs, 
1 =M +m, C12 = U3 + Ms, 


C21 = U2— M4, C22 = U2 + Ms. 

The numerical stability of the recursive version of Winograd’s formula is slightly inferior since the 
error bound of Fact 13 of Section 47.2 holds with y,, = 12n'°® !8, log, 18 © 4.17. 

[Win71], [BD78], [AS81] No algorithm exists for multiplying 2 x 2 matrices with less than 7 
multiplications. The number of nonscalar multiplications needed for multiplying n x n matrices 
is at least 2n? — 1. 


. Ifn = 3*, the matrices A, B, and C canbe partitioned into 9 blocks of size n/3 so that n x n matrix 


multiplication is reduced to computing the product of 3 x 3 block matrices. Formulas for 3 x 3 
matrix multiplication that do not use the commutative property can be recursively used for general 
n X n matrix multiplication. 

In general, if there exists a bilinear noncommutative formula for computing the product of q x q 
matrices with t nonscalar multiplications, then matrices of size n = q* can be multiplied with the 
cost of O(q‘) = O(n'°%: ') arithmetic operations. 

There exist algorithms for multiplying 3 x 3 matrices that require 25 multiplications [Gas71], 
24 multiplications [Fid72], 23 multiplications [Lad76]. None of these algorithms beats Strassen’s 
algorithm since log, 21 < log, 7 < log, 22. No algorithm is known for multiplying 3 x 3 matrices 
with less than 23 nonscalar multiplications. 

[HM73] Rectangular matrix multiplication and the duality property. If there exists a bilinear non- 
commutative algorithm for multiplying two (rectangular) matrices of size n) x np and n2 Xx ns, 
respectively, with ¢ nonscalar multiplications, then there exist bilinear noncommutative algorithms 
for multiplying matrices of size ng, X Ng, and Ng, X Ng, with t nonscalar multiplications for any 
permutation 0 = (0), 02,03). 

If there exists a bilinear noncommutative algorithm for multiplying n, x nz and nz x n3 matrices 
with ¢ nonscalar multiplications, then square matrices of size q = njn2n3 can be multiplied with 
® multiplications. 

From Fact 7 and Fact 4 it follows that if there exists a bilinear noncommutative algorithm for 
multiplying n; x nz and nz x n3 matrices with t nonscalar multiplications, then n x n matrices 
can be multiplied with O(n") arithmetic operations, where w = log,, ,,,,, eb. 

[HK71] There exist bilinear noncommutative algorithms for multiplying matrices of size 2 x 2 and 
2 x 3 with 11 multiplications; there exist algorithms for multiplying matrices of size 2 x 3 and 3 x 3 
with 15 multiplications. 

There are several implementations of fast algorithms for matrix multiplication based on Strassen’s 
formula and on Winograd’s formula. In 1970, R. Brent implemented Strassen’s algorithm on an 
IBM 360/67 (see [Hig02, p. 436]). This implementation was faster than the conventional algorithm 
already for n > 110. In 1988, D. Bailey provided a Fortran implementation for the Cray-2. Fortran 
codes, based on the Winograd variant have been provided since the late 1980s. For detailed com- 
ments and for more bibliography in this regard, we refer the reader to [Hig02, Sect. 23.3]. 
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47.4 Approximation Algorithms 


Matrices can be multiplied faster if we allow that the matrix product can be affected by some arbitrarily 
small nonzero error. Throughout this section, the underlying field F is R or C and we introduce a param- 
eter A € F that represents a nonzero number with small modulus. Multiplication by 4 and A~" is negligible 
in the complexity estimate for two reasons: firstly, by choosing 4 equal to a power of 2, multiplication by 
A can be accomplished by shifting the exponent in the base-two representation of floating point numbers. 
This operation has a cost lower than the cost of multiplication. Secondly, in the block application of matrix 
multiplication algorithms, multiplication by 4 corresponds to multiplying an m x m matrix by the scalar 
2. This operation costs only m? arithmetic operations like matrix addition. 


Definitions: 
Algorithms for multiplying the p x p matrices A and B of the form 


p Pp 
ee Ds Oi, jk Gi, j ye Bije@ig |, kK=1,...t, 


i,j=l i,j=l 
iE 

Cig = 5 Vi,jkIMe + Api,j(A), 
k=1 


where aj, ;,4 Bi,j,k> Vi,j,k ave given rational functions of 4 and p;,;(A) are polynomials, are called Arbitrary 
Precision Approximating (APA) algorithms with ¢ nonscalar multiplications [Bin80], [BCL79]. 


Facts: 
1. [BLR80] The matrix-vector product 


l-l IEI- 


cannot be computed with less than three multiplications. However, the following APA algorithm 
approximates f, and f) with two nonscalar multiplications: 


ax + by 
ay 


m, =(a+Ab)(x+A71y), m, =ay, 
fi ~ fi + dbx = mM, — AT lm, 

fa = mM). 
The algorithm is not defined for A = 0, but for ) — 0 the output of the algorithm converges to 


the exact solution if performed in exact arithmetic. 
2. [BCL79] Consider the 2 x 2 matrix product C = AB where a,> = 0, i.e., 


C11 1,2 i 411 0 bit bi2 
C21 €2,2 42,1 42,2| |b21 b22 : 
The elements c;,; can be approximated with 5 nonscalar multiplications by means of the following 
identities: 
m, = (41,1 + 422)(b12 + Ab21), mz = a4, (bi, + Ab21), m3 = a2,2(b12 + Ab22); 
mg = (41,1 + Aa2,1)(b1,1 — B12), ms = (Aa2,1 — a2,2)bi 2, 
C11 = M2 — hay br, C12 = my — m3 — A(a,,1b2,1 + 2,2b2,1 — a2,2b22), 
C21 =A7'(m, — m2, + mgt ms), C22 = A7'(m3 + ms). 


3. Formulas of Fact 2 can be suitably adjusted to the case where only the element a), of the matrix A 
is zero. 
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4. The product of a 3 x 2 matrix and a2 x 2 matrix can be approximated with 10 nonscalar multi- 
plications by simply combining the formulas of Fact 2 and of Fact 3 in the following way: 


# # 0 O 
=> 0 # + |x 0 | | - 
0 0 x 


5. Facts 4, 7, and 8 of section 47.3 are still valid for APA algorithms. By Fact 7 of Section 47.3 it follows 
that 1000 nonscalar multiplications are sufficient to approximate the product of 12 x 12 matrices. 

6. By Fact 8 of Section 47.3 it follows that O(1'°8:2 1°) arithmetic operations are sufficient to approx- 
imate the product of n x n matrices, where log,, 1000 = 2.7798... < log, 7. 

7. [BLR80] A rounding error analysis of an APA algorithm shows that the relative error in the output 
is bounded by wa" + €6A—*, where a and £ are positive constant depending on the input values, 
h and k are positive constants depending on the algorithm and € is the machine precision. This 
bound grows to infinity as A converges to zero. Asymptotically, in € the minimum bound is ye ™*, 
for a constant y. 

8. [Bin80] From approximate to exact computations. Given an APA algorithm for approximating a 
k x k matrix product with t nonscalar multiplications, there exists an algorithm for n x n exact 
matrix multiplication requiring O(n" log n) arithmetic operations where w = log, t. In particular, 
by Fact 5 an O(n” logn) algorithm exists for exact matrix multiplication with w = log,, 1000. 
This provides the bound w < log,, 1000 for the exponent w of matrix multiplication defined in 
Section 47.2. 

9. The exact algorithm is obtained from the approximate algorithm by applying the approximate 
algorithm with O(log) different values (not necessarily small) of 2 and then taking a suitable 
linear combination of the O(log 1) different values obtained in this way in order to get the exact 
product. More details on this approach, which is valid for any APA algorithm that does not use the 
commutative property of multiplication, can be found in [Bin80]. 


Examples: 


1. The algorithm of Fact 1 computes f\(A) = fi; + Abx that is close to f; for a small lambda. On the 
other hand, one has f; = (fi (1) + fi(—1))/2; Le., a linear combination of the values computed 
by the APA algorithm with 4 = 1 and with A = —1 provides exactly f). 


47.5 Advanced Techniques 


More advanced techniques have been introduced for designing fast algorithms for n x n matrix multi- 
plication. The asymptotically fastest algorithm currently known requires O(n?) arithmetic operations, 
but it is faster than the conventional algorithm only for huge values of n. Finding the infimum w of the 
numbers w for which there exist O(n”) complexity algorithms is still an open problem. In this section, 
we provide a list of the main techniques used for designing asymptotically fast algorithms for n x n matrix 
multiplication. 


Facts: 


1. [Pan78] Trilinear aggregating technique. Different schemes for (approximate) matrix multiplication 
are based on the technique of trilinear aggregating by V. Pan. This technique is very versatile: 
Different algorithms based on this technique have been designed for fast matrix multiplication of 
several sizes [Pan84]; in particular, an algorithm for multiplying 70 x 70 matrices with 143,640 
multiplications which leads to the bound w < 2.795. 
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[Sch81] Partial matrix multiplication. In the expression 
m 
Gj = SS aby: i,j=l,...,m, 
r=1 


there are m> terms which are summed up. A partial matrix multiplication is encountered if some 
aj,j or some b;,; are zero, or if not all the elements c;,; are computed so that in the above expression 
there are less than m? terms, say, k < m.A. Schénhage [Sch81] has proved that if there exists a 
noncommutative bilinear (APA) algorithm for computing (approximating) partial m x m matrix 
multiplication with k terms that uses t nonscalar multiplications, then w < 3 log, t. This result 
applied to the formula of Fact 2 of Section 47.4 provides the bound w < 3 log, 5 < 2.695. 

[Sch81] Disjoint matrix multiplication A. Schénhage has proven that it is possible to approximate 
two disjoint matrix products with less multiplications than the number of multiplications needed 
for computing separately these products. In particular, he has provided an APA algorithm for 
simultaneously multiplying a 4 x 1 matrix bya 1 x 4anda1 x 9 matrix by a9 x 1 with only 17 
nonscalar multiplications. Observe that 16 multiplications are needed for the former product and 
9 multiplications are needed for the latter. 

[Sch81] The t-theorem. A. Schénhage has proven the t-theorem. Namely, if the set of disjoint 
matrix products of size m; x nj; times nj x pj, fori = 1,...,k can be approximated with t 
nonscalar multiplications by a bilinear noncommutative APA algorithm, then w < 3t, where t 
solves the equation pee (mn; pi)’ = t. From the disjoint matrix multiplication of Fact 3 it follows 
the equation 167 + 9° = 17, which yields w < 3t = 2.5479.... 


. [Str87], [Str88] Asymptotic spectrum: the laser method. In 1988, V. Strassen introduced a powerful 


and sophisticated method, which, by taking tensor powers of set of bilinear forms that appar- 
ently are not completely related to matrix multiplication, provides some scheme for fast matrix 
multiplication. The name laser method was motivated from the fact that by tensor powering a set 
of “incoherent” bilinear forms it is possible to obtain a “coherent” set of bilinear forms. 

Lower bounds. At least 2n? — 1 multiplications are needed for multiplying n x n matrices by 
means of noncommutative algorithms [BD78]. If n > 3, then nr +tn—2 multiplications are 
needed [Bla03]. The lower bound Sn? — 3n has been proved in [Bla01]. The nonlinear asymptotic 
lower bound n? log n has been proved in [Raz03]. At least n? + 2n — 2 multiplications are needed 
for approximating the product of n x n matrices by means of a noncommutative APA algorithm 
[Bin84]. The lower bound turns to n* + an — 2 multiplications if commutativity is allowed. The 
product of 2 x 2 matrices can be approximated with 6 multiplications by means of a commutative 
algorithm [Bin84], 5 multiplications are needed. Seven multiplications are needed for approximat- 
ing 2 x 2 matrix product by means of noncommutative algorithms [Lan05]. 

History of matrix multiplication complexity. After the 1969 paper by V. Strassen [Str69], where it 
was shown that O(n!°8:”) operations were sufficient for n x n matrix multiplication and inver- 
sion, the exponent w of matrix multiplication complexity remained stuck at 2.807... for almost 
10 years until when V. Pan, relying on the technique of trilinear aggregating, provided a bilinear 
noncommutative algorithm for 70 x 70 matrix multiplication using 143,640 products. This led 
to the upper bound w < log, 143640 ~ 2.795. A few months later, Bini, Capovani, Lotti, and 
Romani [BCL79] introduced the concept of APA algorithms, and presented a scheme for approx- 
imating a 12 x 12 matrix product with 1000 products. In [Bin80] Bini showed that from any APA 
algorithm for matrix multiplication it is possible to obtain an exact algorithm with almost the same 
asymptotic complexity. This led to the bound w < log,, 1000 © 2.7798. The technique of partial 
matrix multiplication was introduced by Schénhage [Sch81] in 1981 together with the t-theorem, 
yielding the bound w < 2.55, a great improvement with respect to the previous estimates. This 
bound relies on the tools of trilinear aggregating and of approximate algorithms. Based on the 
techniques so far developed, V. Pan obtained the bound w < 2.53 in [Pan80] and one year later, 
F. Romani obtained the bound w < 2.52 in [Rom82]. The landmark bound w < 2.5 was obtained 
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47.6 


TABLE 47.1 Main steps in the history of fast matrix multiplication 
o< 
2.81 1969 Str69] Bilinear algorithms 
2.79 1979 Pan78 Trilinear aggregating 
2.78 1979 BCL79],{Bin80] | Approximate algorithms 
2.55 1981 Sch81 t-theorem 
2.53 1981 Pan80 
2.92 1982 Rom82] 
2.50 1982 CW72 Refinement of the t-theorem 
2.48 1987,1988 Str87],[Str88] Laser method 
2.38 1990 CW82 


by Coppersmith and Winograd [CW72] in 1982 by means of a refinement of the t-theorem. In 
[Str87] Strassen introduced the powerful laser method and proved the bound w < 2.48. The laser 
method has been perfected by Coppersmith and Winograd [CW82], who proved the best esti- 
mate known so far, i.e., @ < 2.38. Table 47.1 synthesizes this picture together with the main con- 
cepts used. 


Applications 


Some of the main applications of matrix multiplication are outlined in this section. For more details the 
reader is referred to [Pan84]. 


Definitions: 


A square matrix A is strongly nonsingular if all its principal submatrices are nonsingular. 


Facts: 
1. 


Classic matrix inversion. Given a nonsingular n x n matrix A, the elements of A! can be computed 
by means of Gaussian elimination in O(n3) arithmetic operations. 

Inversion formula. Let n = 2m and partition the n x n nonsingular matrix A into four square blocks 
of size m as 


Aji Ai 
Az A2,2 


Assume that A; is nonsingular. Denote S = Ay. — Ao, Ay} Ai2 the Schur complement of A; in 
Aand R = Aj}. Then S is nonsingular and the inverse of A can be written as 


aa RE RASS AAR HRA So 
a6 AG s7 


Moreover, det A = det S det Aj). 

Fast matrix inversion. Let n = 2*, with k positive integer and assume that A is strongly nonsingular. 
Then also Aj, and the Schur complement S are strongly nonsingular and the inversion formula of 
Fact 2 can be applied again to S by partitioning S into four square blocks of size n/4, recursively 
repeating this procedure until the size of the blocks is 1. The algorithm obtained in this way 
is described in Algorithm 3. Denoting by Z(n) the complexity of this algorithm for inverting a 
strongly nonsingular n x n matrix and denoting by M(n) the complexity of the algorithm used 
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for n x n matrix multiplication, we obtain the expression 


Z(n) = 2T(n/2) + 6M(n/2) + n?/2, 

Ly =1; 
If M(n) = O(n") with w > 2, then one deduces that Z(n) = O(n”). That is, the complexity of 
matrix inversion is not asymptotically larger than the complexity of matrix multiplication. 


4, The complexity of matrix multiplication is not asymptotically greater than the complexity of matrix 
inversion. This property follows from the simple identity 


-1 


I A O I -A AB 
0 I B =|]0 I -B 
0 0 1 0 0 I 


5. Combining Facts 3 and 4, we deduce that matrix multiplication and matrix inversion have the same 
asymptotical complexity. 


Algorithm 3: Fast matrix inversion 


Procedure Fast_Inversion(k, A) 
Input: the elements a;,; of the 2k x 2k strongly nonsingular matrix A; 
Output: the elements b;,; of B = A7!. 


Ifk = 0, then 
output by.) = ay ]3 
else 


partition A, into 2‘! x 2*~! blocks A 
set R = Fast_Inversion(k — 1, Aj,1); 
compute S = Aj» — Az; RAj2 and V = Fast_Inversion(k — 1, S) 
output 
RRA VAGR =RA GV 
AVAGGR V 


ijots J = 1, 2; 


6. The property of strong singularity for A is not a great restriction if F = R or F = C. In fact, if A 
is nonsingular then A~' = (A*A)~! A* and the matrix A* A is strongly nonsingular. 

7. Computing the determinant. Let A be strongly nonsingular. From Fact 2, det A = det S det Aj). 
Therefore, an algorithm for computing det A can be recursively designed by computing S and then 
by applying recursively this algorithm to S and to A,, until the size of the blocks is 1. Denoting by 
D(n) the complexity of this algorithm one has 


D(n) = 2D(n/2) + Z(n/2) + 2M (n/2) + n°/4, 


hence, if M(n) = O(n") with w > 2, then D(n) = O(n"). In [BS83] it is shown that M(n) = 
O(D(n)) 

8. Computing the characteristic polynomial. The coefficients of the characteristic polynomial p(x) = 
det(A — xI) of the matrix A can be computed with the same asymptotic complexity of matrix 
multiplication. 

9. Combinatorial problems. The complexity of some combinatorial problems is related to matrix 
multiplication, in particular, the complexity of the all pair shortest distance problem of finding the 
shortest distances d(i, k) from i to k for all pairs (i,k) of vertices of a given digraph. We refer the 
reader to section 18 of [Pan84] for more details. The problem of Boolean matrix multiplication 
can be reduced to that of general matrix multiplication. 
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48.1 Structured Matrices 


In various application fields, the matrices encountered have special structures that can be exploited to 
facilitate the solution process. Sparsity is one of these features. However, the matrices we consider in this 
chapter are mostly dense matrices with a special structure. Structured matrices have been around for a long 
time and are encountered in various fields of application. (See [GS84, KS98, BTY01].) Some interesting 
families are listed below. For simplicity, we give the definitions for real square matrices of size n. 


Definitions: 


Toeplitz matrices: Matrices with constant diagonals, i.e., [T]j,; = 4; for alll <i,j <m: 


ti—2 


ti-1 


ty bn ben 
tf  t-1 bn 
t to 
ty} 
tn—-2 ty to 


(See Chapter 16.2 for additional information on families of Toeplitz matrices.) 
Lower shift matrix: The matrix with ones on the first subdiagonal and zeros elsewhere: 


0 0 0 O 

1 0 0 0 
1 0 

0 

0 1 0 
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Circulant matrices: Toeplitz matrices where each column is a circular shift of its preceding column: 


Co Ch-1 ot" C2 C) 
Cl Co Cn-1 C2 
C2 cy Co 

C = 
Cn—2 Cnh-1 
Cn-1 Cn-2 7°" C2 Cy Co 


n-Cycle matrix: The n x n matrix with ones along the subdiagonal and in the 1, n-entry, and zeros 
elsewhere, i.e., 


0 Made 0) 
1 0 0 
Ch = i: <6 
0 0 
0 0 1 0 


ho hy hy-2 hina 

hy hy . hn 

St 2. ae 
higsgs eh Pa od hon—2 
Ry An +t) Man-2 Moni 


0 0 0 1 
0 0 1 O 
P, = 
0 1 0 
0 0 O 
Cauchy matrices: Given vectors x = [x),... Xn]? andy = [y1,...,¥n]", the Cauchy matrix C(x, y) 


1 
xityj* 
Vandermonde matrices: A matrix having each row equal to successive powers of a number, Le., 


has i, j-entry equal to 


1 n—2 n—-1 
lvoe Wy V1 
1 n—2 n—-1 
Lo vy t+) Vy Vy 
V= . 
1 yl yn-2 pri 
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Block matrices: An m x m block matrix with n x n blocks is a matrix of the form 


AG) Alb?) eS Albm 
AD Al2?) sates Alam) 
Al™@-LD  Alm—1,2) 62. A (mm) 
Alm) Alm2) Js Alm) 


where each block A“) is an n x n matrix. 

Toeplitz-block matrices: mn x mn block matrices where each block {A“/)} 
matrix. 

Block-Toeplitz matrices: mn x mn block matrices of the form 


m 


ij=1 is ann x n Toeplitz 


A) ACY ate Au-m 
AW A) we. Al2—m) 
T= ? : ge |5 
Aon) Alt?) eh A) 


where {A@yaL m are arbitrary n x n matrices. 
Block-Toeplitz—Toeplitz-block (BTTB) matrices: The blocks A“) are themselves Toeplitz matrices. 
Block matrices for other structured matrices such as the block-circulant matrices or the circulant-block 


matrices can be defined similarly. 


Facts: 


1. The transpose of a Toeplitz matrix is a Toeplitz matrix. 
. Any linear combination of Toeplitz matrices is a Toeplitz matrix. 
. The lower shift shift matrices Z,, are Toeplitz matrices. The n x n Toeplitz matrix T = [#;] with 
tj = t-j satisfies T = foI,, + nets + Wetey(Ze 
. Every circulant matrix is a Toeplitz matrix, but not conversely. 
. The transpose of a circulant matrix is a circulant matrix. 
. Any linear combination of circulant matrices is a circulant matrix. 
The n-cycle matrix C,, is a circulant matrix. The n x n circulant matrix C = [c;;] with cj; = cj-1 
satisfies C = ee Oe 
8. An important property of circulant matrices is that they can diagonalized by discrete Fourier 
transform matrices (see Section 47.3 and Chapter 58.3). Thus circulant matrices are normal. 
9. A Hankel matrix is symmetric. 
10. Any linear combination of Hankel matrices is a Hankel matrix. 
11. Multiplication of a Toeplitz matrix and the anti-identity matrix P,, is a Hankel matrix, and multi- 
plication of a Hankel matrix and P,, is a Toeplitz matrix. For H as in the definition, 


Ww N 


NOU B® 


hn-1 ln tee Aop-2 Mani 
a) hy-1 hy hon—2 
P,H = Ayr Nyy > 
hy hy 
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Ay-1 hy—2 eal hy ho 
hy, Ay-1 hy-2 hy 
AP, = hy hy 
hon—2 See: Was Sieg 
hon—1 hon—2 MON hy hn 


12. The Kronecker product A ® T of any matrix A and a Toeplitz matrix T is a Toeplitz-block matrix 
and Kronecker product T ® A is a block-Toeplitz matrix. 

13. Most of the applications, such as partial differential equations and image processing, are concerned 
[Jin02], [Ng04] with two-dimensional problems where the matrices will have block structures. 


Examples: 
1 2 1 1 
aes ie a & eich aaees peers 11 5.2 
i isa BTTB matrix with A® = AY = AY = 
—-5 2 1 2 3 1 1 0 5 
0 —-5 3 1 
1 1 1 
| 1 ae 
2. Forx = |1] andy= |2},C(xy) = ; ; t 
1 1 1 
3 6 7 8 


48.2 Direct Toeplitz Solvers 


Most of the early work on Toeplitz solvers was focused on direct methods. These systems arise in a variety 
of applications in mathematics and engineering. In fact, Toeplitz structure was one of the first structures 
analyzed in signal processing. 


Definitions: 


Toeplitz systems: A system of linear equations with a Toeplitz coefficient matrix. 


Facts: 


1. Given an n x n Toeplitz system Tx = b, a straightforward application of the Gaussian elimination 
method will result in an algorithm of O(n*) complexity. 

2. However, since the matrix is determined by only (2n — 1) entries rather than n? entries, it is to be 
expected that a solution can be obtained in less than O(n) operations. There are a number of fast 
Toeplitz solvers that can reduce the complexity to O(n”) operations. The original references for 
these algorithms are Schur [Sch17], Levinson [Lev46], Durbin [Dur60], and Trench [Tre64]. 

3. In the 1980s, superfast algorithms of complexity (nlog* 1) operations for Toeplitz systems were 
proposed by a different group of researchers: Bitmead and Anderson [BA80], Brent et al. [BGY80], 
Morf [Mor80], de Hoog [Hoo87], and Ammar and Gragg [AG88]. The key to these direct methods 
is to solve the system recursively. In this section, we will give a brief summary of the development 
of these methods. We refer the reader to the works cited for more details. 

4. If the Toeplitz matrix T has a singular or ill-conditioned principal submatrix, then a breakdown 
or near-breakdown can occur in the direct Toeplitz solvers. Such breakdowns will cause numerical 
instability in subsequent steps of the algorithms and result in inaccurately computed solutions. The 
question of how to avoid breakdowns or near-breakdowns by skipping over singular submatrices 
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or ill-conditioned submatrices has been studied extensively, and various such algorithms have been 
proposed. (See Chan and Hansen [CH92].) 

5. The fast direct Toeplitz solvers are in general numerically unstable for indefinite systems. Look- 
ahead methods are numerically stable and, although it may retain the O(n”) complexity, it requires 
O(n?) operations in the worst case. 

6. The stability properties of direct methods for symmetric positive definite Toeplitz systems were 
discussed in Sweet [Swe84], Bunch [Bun85], Cybenko [Cyb87], and Bojanczyk et al. [BBH95]. 

7. Gohberg et al. [GKO95] have shown how to perform Gaussian elimination in a fast way for matrices 
having special displacement structures. Such matrices include Toeplitz, Vandermonde, Hankel, and 
Cauchy matrices. They have shown how to incorporate partial pivoting into Cauchy solvers. They 
pointed out that although pivoting cannot be incorporated directly for Toeplitz matrices, Toeplitz 
problems can be transformed by simple orthogonal operations to Cauchy problems. The solutions 
to the original problems can be recovered from those of the transformed systems by the reverse 
orthogonal operations. Thus, fast Gaussian elimination with partial pivoting can be carried out for 
Toeplitz systems. 

Brent and Sweet [BS95] gave a rounding error analysis on the Cauchy and Toeplitz variants of the 
recent method of Gohberg et al. [GK095]. It has been shown that the error growth depends on the 
growth in certain auxiliary vectors, the generators, which are computed by the Gohberg algorithm. 
In certain circumstances, growth in the generators can be large, so the error growth is much larger 
than would be encountered when using normal Gaussian elimination with partial pivoting. 


48.3 Iterative Toeplitz Solvers 


A circulant matrix is a special form of Toeplitz matrix where each row of the matrix is a circular shift of its 
preceding row. Because of the periodicity, circulant systems can be solved quickly via a deconvolution by 
discrete Fast Fourier Transforms (FFTs) [Ng04]. Circulant approximations to Toeplitz matrices have been 
used for some time in signal and image processing [Ng04]. However, in these applications, the circulant 
approximation so obtained is used to replace the given Toeplitz matrix in subsequent computations. In 
effect, the matrix equation is changed and, hence, so is the solution. 

Development of solely circulant-based iterative methods for Toeplitz systems started in the 1970s. Rino 
[Rin70] developed a method for generating a series expansion solution to Toeplitz systems by writing a 
Toeplitz matrix as the sum of a circulant matrix and another Toeplitz matrix and presented a method for 
choosing the circulant matrix. Silverman and Pearson [SP73] applied similar methods to deconvolution. 

In 1986, Strang [Str86] and Olkin [Olk86] independently proposed to precondition Toeplitz matrices 
by circulant matrices in conjugate gradient iterations. Their motivation was to exploit the fast inversion of 
circulant matrices. Numerical results in [SE87] and [Olk86] show that the method converges very fast for a 
wide range of Toeplitz matrices. This has later been proved theoretically in [CS89] and in other papers for 
other circulant preconditioners [Ng04]. Circulant approximations are used here only as preconditioners 
for Toeplitz systems and the solutions to the Toeplitz systems are unchanged. 

One of the main important results of this methodology is that the complexity of solving a large class of 
n x n Toeplitz systems can be reduced to O(n log n) operations, provided that a suitable preconditioner 
is used. Besides the reduction of the arithmetic complexity, there are important types of Toeplitz matrix 
where the fast direct Toeplitz solvers are notoriously unstable, e.g., indefinite and certain non-Hermitian 
Toeplitz matrices. Therefore, iterative methods provide alternatives for solving these Toeplitz systems. 


48.4 Linear Systems with Matrix Structure 


This section provides some examples of the latest developments on iterative methods for the itera- 
tive solution of linear systems of equations with structured coefficient matrices such as Toeplitz-like, 
Toeplitz-plus-Hankel, and Toeplitz-plus-band matrices. We would like to make use of their structure to 
construct some good preconditioners for such matrices. 


48-6 


Facts: 
1. 


Handbook of Linear Algebra 


Toeplitz-like systems: Let A be an n x n structured matrix with respect to Z,, (the lower shift matrix): 
VA, = A, — Z,A,Z* = GSG* 


for some n x r generator matrix G andr x r signature matrix S = (I, © —I,). If we partition the 
: p-l q-1 
columns of G into two sets {x;};_) and {yi};_9> 


G = [x x1... Xp-1 oN». Yg-1] with p+q =r, 


then we know from the representation that we can express A as a linear combination of lower 
triangular Toeplitz matrices, 


pol at 
A= S_L(x)L*(xi;) — > LU )L* (4). 
i=0 i=0 


For example, if T;,,, isan m x n Toeplitz matrix with m > n, then T; ,, Tin isin general not a Toeplitz 
matrix. However, Ty ., Tinn does have a small displacement rank, r < 4, and the displacement 
representation of T |, Tinn is 

dia Lian a Ly (xo) Ln(x0)* = Li(yo)Ln(yo)* + Ly(x1)Ln(x1)* -, Li(yi)La(n)*, 


where 


x0 = Tn Tmnei/l[Tmneill yo = ZnZ;,Xos 


x)= [0, pain ky Si eae Chesil and n= [0, tin—15 tm—29°°* ytm—ntil*. 


. For structured matrices with displacement representations, it was suggested in [CNP94] to define 


the displacement preconditioner to be the circulant approximation of the factors in the displace- 
ment representation of A, i.e., the circulant approximation C of A is 


pol q-1 
S 5 C(L(x;))C* (Lux) — $5 CLa(yi))C* Ln). 
i=0 i=0 


Here, C(X) denotes some circulant approximations to X. 


. The displacement preconditioner approach is applied to Toeplitz least squares and Toeplitz-plus- 


Hankel least squares problems [Ng04]. 


. The systems of linear equations with Toeplitz-plus-Hankel coefficient matrices arise in many signal 


processing applications. For example, the inverse scattering problem can be formulated as Toeplitz- 
plus-Hankel systems of equations (see [Ng04].) 
The product of P,, and H and the product of H and P,, both give Toeplitz matrices. 
Premultiplying P,, to a vector v corresponds to reversing the order of the elements in v. Since 


Hv = HP, Priv 


and H P,, is a Toeplitz matrix, the Hankel matrix-vector products Hv can be done efficiently using 
FFTs. A Toeplitz-plus-Hankel matrix can be expressed as T+ H = T + P,P, H. 


. Given circulant preconditioners C and C® for Toeplitz matrices T and P,, H, respectively, it was 


proposed in [KK93] to use 


M=C" + P,c” 
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as a preconditioner for the Toeplitz-plus-Hankel matrix T + H. With the equality P? = I, we have 


Mz=C%z+ P,C? P,, P,z =v, 
which is equivalent to 
P,Mz = P,C™ P,P,z+C%z = Pry. 


By using these two equations, the solution of z = M~'v can be determined. 

6. We consider the solution of systems of the form (T + B)x = b, where T is a Toeplitz matrix and 
B is a banded matrix with bandwidth 2b + 1 independent of the size of the matrix. These systems 
appear in solving Fredholm integro-differential equations of the form 


B 
L{x()} + / K($ —0)x($)do = b(6). 


Here, x(@) is the unknown function to be found, K (9) is a convolution kernel, L is a differential 
operator and b(@) is a given function. After discretization, K will lead to a Toeplitz matrix, L to 
a banded matrix, and b(@) to the right-hand side vector [Ng04]. Toeplitz-plus-band matrices also 
appear in signal processing literature and have been referred to as periheral innovation matrices. 

Unlike Toeplitz systems, there exist no fast direct solvers for solving Toeplitz-plus-band systems. 
It is mainly because the displacement rank of the matrix T + B can take any value between 0 and 
n. Hence, fast Toeplitz solvers that are based on small displacement rank of the matrices cannot 
be applied. Conjugate gradient methods with circulant preconditioners do not work for Toeplitz- 
plus-band systems either. The main reason is that when the eigenvalues of B are not clustered, the 
matrix C(T) + B cannot be inverted easily. 

In [CN93], it was proposed to use the matrix E + B to precondition T + B, where E is the 
band-Toeplitz preconditioner such that E is spectrally equivalent to T. Note that E is a banded 
matrix, and the banded system (E + B)y = z can be solved by using any band matrix solver. 

7. Banded preconditioners are successfully applied to precondition Sinc—Galerkin systems (Toeplitz- 
plus-band systems) arising from the Sinc—Galerkin method to partial differential equations (see 
[Ng04].) 

8. In most of applications we simply use the circulant or other transform-based preconditioners 
[Ng04]. We can extend the results for point circulant or point transform-based preconditioners 
to block circulant preconditioners or block transform-based preconditioners [Jin02] for block- 
Toeplitz, Toeplitz-block, and Toeplitz-block—block-Toeplitz matrices. 

9. Consider the system (A @ B)x = b, where A is an m-by-m Hermitian positive definite matrix 
and B is an n-by-n Hermitian positive definite Toeplitz matrix. By using a circulant approximation 
C(B) to B, the preconditioned system becomes 


(A @ C(B))"'(A®@ B)x = (A@C(B))'b, 
or 
(P@.C(B) "Bx =(A" @ C(BY")b. 


When B is a Hermitian positive definite Toeplitz matrix, C(B) can be obtained in O(n) operation 
[Jin02], [Ng04]. The initialization cost is about O(m? + m?n + mn logn) operations. Moreover, 
since the cost of multiplying By becomes O(n log n), we see that the cost per iteration is equal to 
O(mn log n) when iterative methods are employed. 

10. When Toeplitz matrices have full rank, Toeplitz least squares problems 


min || Tx — || 
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are equivalent to solving the normal equation matrices 
T*Tx = T*b. 


Circulant preconditioners can be applied effectively and efficiently to solving Toeplitz least squares 
problems if Toeplitz matrices have full rank. When Toeplitz matrices do not have full rank, it 
is still an open research problem to find efficient algorithms for solving rank-deficient Toeplitz 
least squares problem. One possibility is to consider the generalized inverses of Toeplitz matrices. 
In the literature, computing the inverses and the generalized inverses of structured matrices are 
important practical computational problems. (See, for instance, Pan and Rami [PRO1] and Bini 
et al. [BCV03].) 

Instead of Toeplitz least squares problems min || Tx — b||>, we are interested in the 1-norm problem, 
ie., min || Tx — b||;. The advantage of using the 1-norm is that the solution is more robust than 
using the 2-norm in statistical estimation problems. In particular, a small number of outliers have 
less influence on the solution. It is interesting to develop efficient algorithms for solving 1-norm 
Toeplitz least squares problems. Fu et al. [FNN06] have considered the least absolute deviation 
(LAD) solution of image restoration problems. 

It is interesting to find good preconditioners for Toeplitz-related systems with large displacement 
rank. Good examples are Toeplitz-plus-band systems studied. Direct Toeplitz-like solvers cannot be 
employed because of the large displacement rank. However, iterative methods are attractive since 
coefficient matrix—vector products can be computed efficiently at each iteration. For instance, for 
the Toeplitz-plus-band matrix, its matrix-vector product can be computed in O(n log n) operations. 
The main concern is how to design good preconditioners for such Toeplitz-related systems with 
large displacement rank. Recently, Lin et al. [LNC05] proposed and developed factorized banded 
inverse preconditioners for matrices with Toeplitz structure. Also, Lin et al. [LCN04] studied 
incomplete factorization-based preconditioners for Toeplitz-like systems with large displacement 
ranks in image processing. 


48.5 Total Least Squares Problems 


1. 


The least squares problem Tf ~ g is 
mam ITF — gle. 


If the matrix T is known exactly, but the vector g is corrupted by random errors that are un- 
correlated with zero mean and equal variance, then the least squares solution provides the best 
unbiased estimate of f. However, if T is also corrupted by errors, then the total least squares (TLS) 
method may be more appropriate. The TLS problem minimizes 


min ||[T g] — (7 gllli 
Ug 
with the constraint T f = g. If the smallest singular value of T is larger than the smallest singular 


value o” of [T g], then there exists a unique TLS solution f7,5, which can be represented as the 
solution to the normal equations: 


(T’T —o7I)f =T’g, 


or as the solution to the eigenvalue problem: 


T'T Tg| | f Sick 
gtr gt} |-1}° [-ay 
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Kamm and Nagy [KN98] proposed using Newton and Rayleigh quotient iterations for large TLS 
Toeplitz problems. Their method is a modification of a method suggested by Cybenko and Van 
Loan [CV86] for computing the minimum eigenvalue of a symmetric positive definite Toeplitz 
matrix. Specifically, first note that the TLS solution fr, 5 solves the eigenvalue problem. Moreover, 
this eigenvalue problem is equivalent to 


Tore - T'g =o’ f 
and 


g' Tf —g'g =-o’, 


which can be combined to obtain the following secular equation for 07: 
g'g—giT(T'T —o7I)'T'g—o* =0. 
Therefore, o? is the smallest root of the rational equation 
h(a”) = g'g —g'T(T'T — 071) 'T'g —o? 


and can be found using Newton’s method. Note that if o? is less than the smallest singular value 
6? of T, then the matrix T’ T — 071 is positive definite. Assume for now that the initial estimate 
is within the interval [o7, 67). An analysis given in [CV86] shows that subsequent Newton iterates 
will remain within this interval and will converge from the right to 0”. 

2. In the above computation, T is not necessary to have Toeplitz structure. In another development, 
Ng [NPP00] presented an iterative, regularized, and constrained total least squares algorithm by 
requiring T to be Toeplitz. Preliminary numerical tests are reported on some simulated optical 
imaging problems. The numerical results showed that the regularized constrained TLS method is 
better than the regularized least squares method. 

3. Other interesting areas are to design efficient algorithms based on preconditioning techniques 
for finding eigenvalues and singular values of Toeplitz-like matrices. Ng [Ng00] has employed 
preconditioned Lanczos methods for the minimum eigenvalue of a symmetric positive definite 
Toeplitz matrix. 
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Computational problems, especially in science and engineering, often involve large matrices. Examples 
of such problems include large sparse systems of linear equations [FGN92],[Saa03],[vdV03], e.g., arising 
from discretizations of partial differential equations, eigenvalue problems for large matrices [BDD00], 
[LM05], linear time-invariant dynamical systems with large state-space dimensions [FF94],[FF95],[Fre03], 
and large-scale linear and nonlinear optimization problems [KR91],[Wri97],[NW99],[GMS05]. The large 
matrices in these problems exhibit special structures, such as sparsity, that can be exploited in computational 
procedures for their solution. Roughly speaking, computational problems involving matrices are called 
“large-scale” if they can be solved only by methods that exploit these special matrix structures. 

In this section, as in Chapter 44, multiplication of a vector v by a scalar 4 is denoted by vA rather 
than Av. 


49.1 Basic Concepts 


Many of the most efficient algorithms for large-scale matrix computations are based on approximations 
of the given large matrix by small matrices obtained via Petrov—Galerkin projections onto suitably chosen 
small-dimensional subspaces. In this section, we present some basic concepts of such projections. 


Definitions: 
LetC € C"*" andlet Vj =[vi vw. ++: wil é€ C"*/ be a matrix with orthonormal columns, i.e., 
0 ifif¢k, . , 
VivE = a forall i,k = 1,2,...,j. 
1 ifi=k, 
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The matrix 


Cj = V;CV; eC 


is called the orthogonal Petrov—Galerkin projection of C onto the subspace 


S = span{v1,V2,...,Vj} 


of C” spanned by the columns of Vj. 
Let C € C"™", andlet Vj = [v. vw. --- vj] € C"*/ and Wj =[w, wo --- wj] € C"*! be 
two matrices such that wi V; is nonsingular. The matrix 


Cy = (WIV) ‘WiC; eC 


is called the oblique Petrov—Galerkin projection of C onto the subspace 


S = span{v1,V2,...,Vj} 


of C” spanned by the columns of V; and orthogonally to the subspace 


T = span{ Ww), W2,..., Wj} 


of C" spanned by the columns of Wj. 

A flop is the work associated with carrying out any one of the elementary operations a + b, a — b, ab, 
or a/b, where a, b € C, in floating-point arithmetic. 

Let A = [aix] € C’”*” bea given matrix. Matrix-vector multiplications with A are said to be fast if for 
any x € C”, the computation of y = Ax requires significantly fewer than 2mn flops. 


A matrix A = [aj] € C’"*" is said to be sparse if only a small fraction of its entries aj are nonzero. 


For a sparse matrix A = [ajz] € C'*", nnz(A) denotes the number of nonzero entries of A. 


A matrix A = [aj] € C”*" is said to be dense if most of its entries aj, are nonzero. 


Facts: 


The following facts on sparse matrices can be found in [Saa03, Chap. 3] and the facts on computing 
Petrov—Galerkin projections of matrices in [Saa03, Chap. 6]. 


. Forasparse matrix A = [aj] € C”””, only its nonzero or potentially nonzero entries aj, together 


with their row and column indices i and k, need to be stored. 

Matrix-vector multiplications with a sparse matrix A = [aj] € C””” are fast. More precisely, for 
any x € C”, y = Ax can be computed with at most 2nnz(A) flops. 

IfC € C™” and j < n, the computational cost for computing the orthogonal Petrov—Galerkin 
projection of C onto the j-dimensional subspace S = span{ vi, ¥v2,...,vj} of C” is dominated by 
the j matrix-vector products yj; = Cvj,i = 1,2,...,7. 

If C € C"*" and j < n, the computational cost for computing the oblique Petrov—Galerkin 
projection of C onto the j-dimensional subspace S = span{ v1, V2,...,Vvj} of C” and orthogonally 
to the j-dimensional subspace T = span{ Ww), W2,...,wj} of C” is dominated by the j matrix- 
vector products yj = Cvj,i = 1,2,...,j. 

If matrix-vector products with a large matrix C € C”*" are fast, then orthogonal and oblique 
Petrov—Galerkin projections C; of C can be generated with low computational cost. 
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49.2 Sparse Matrix Factorizations 


In this section, we present some basic concepts of sparse matrix factorizations. A more detailed description 
can be found in [DER89]. 


Definitions: 


Let A € C”*" bea sparse nonsingular matrix. A sparse L U factorization of A is a factorization of the form 
A = PLUQ, 


where P, Q € R”*" are permutation matrices, L € C"*" is a sparse unit lower triangular matrix, and 
U € C”*" is a sparse nonsingular upper triangular matrix. 

Fill-in of a sparse LU factorization A = PLUQ is the set of nonzero entries of L and U that appear in 
positions (i,k) where aj, = 0. 

Let A = A* € C"*", A > 0, be a sparse Hermitian positive definite matrix. A sparse Cholesky 
factorization of A is a factorization of the form 


A=PLL*P!, 


where P € R"™” is a permutation matrix and L € C”*" is a sparse lower triangular matrix. 

Fill-in of a sparse Cholesky factorization A = PLL* P? is the set of nonzero entries of L that appear in 
positions (i,k) where az = 0. 

Let T € C"*” be a sparse nonsingular (upper or lower) triangular matrix, and let b € C”. A sparse 
triangular solve is the solution of a linear system 


Tx=b 


with a sparse triangular coefficient matrix T. 


Facts: 
The following facts can be found in [DER89]. 


1. The permutation matrices P and Q ina sparse LU factorization of A allow for reorderings of the 
rows and columns of A. These reorderings serve two purposes. First, they allow for pivoting for 
numerical stability in order to avoid division by the number 0 or by numbers close to 0, which 
would result in breakdowns or numerical instabilities in the procedure used for the computation 
of the factorization. Second, the reorderings allow for pivoting for sparsity, the goal of which is to 
minimize the amount of fill-in. 

2. For Cholesky factorizations of matrices A = A* > 0, the positive definiteness of A implies that 
pivoting for numerical stability is not needed. Therefore, the permutation matrix P in a sparse 
Cholesky factorization serves the single purpose of pivoting for sparsity. 

3. For both sparse LU and sparse Cholesky factorizations, the problem of “optimal” pivoting for spar- 
sity, Le., finding reorderings that minimize the amount of fill-in, is NP-complete. This means that 
for practical purposes, minimizing the amount of fill-in of factorizations of large sparse matrices 
is impossible in general. However, there are a large number of pivoting strategies that — while 
not minimizing fill-in — efficiently limit the amount of fill-in for many important classes of large 
sparse matrices. (See, e.g., [DER89].) 

4. A sparse triangular solve with the matrix T requires at most 2nnz(T) flops. 

5. Not every large sparse matrix A has a sparse LU factorization with limited amounts of fill-in. For 
example, LU or Cholesky factorizations of sparse matrices A arising from discretization of partial 
differential equations for three-dimensional problems are often prohibitive due to the large amount 
of fill-in. 
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Examples: 


1. Given a sparse LU factorization A = PLUQ ofa sparse nonsingular matrix A € C"*", the solution 
x of the linear system Ax = b with any right-hand side b € C” can be computed as follows: 


Set c= Pb, 
Solve Lz=c for z, 
Solve Uy=z for y, 
Set x= Q’y. 


Since P and Q are permutation matrices, the first and the last steps are just reorderings of the entries 
of the vectors b and y, respectively. Therefore, the main computational cost is the two triangular 
solves with L and U, which requires at most 2(nnz(L) + nnz(U)) flops. 

2. Given a sparse Cholesky factorization A = PLL* P? of a sparse Hermitian positive definite matrix 
A € C”*", the solution x of the linear system Ax = b with any right-hand side b € C” can be 
computed as follows: 


Set c= P?b, 
Solve Lz=c for z, 
Solve L*y=z for y, 
Set x= P'y. 


Since P is a permutation matrix, the first and the last steps are just reorderings of the entries of the 
vectors b and y, respectively. Therefore, the main computational cost is the two triangular solves 
with L and L*, which requires at most 4nnz(L) flops. 

3. In large-scale matrix computations, sparse factorizations are often not applied to a given sparse 
matrix A € C"*”, but to a suitable “approximation” Ay € C"*” of A. For example, if sparse 
factorizations of A itself are prohibitive due to excessive fill-in, such approximations Ap can often 
be obtained by computing an “incomplete” factorization of A that simply discards unwanted fill-in 
entries. Given a sparse LU factorization 


Ao = PLUQ 


of a sparse nonsingular matrix Ap € C”*”, which in some sense approximates the original matrix 
A € C”*", one then uses iterative procedures that only involve matrix-vector products with the 
matrix 


C := Ap'A= Q’U"'L"'P'A, 


or possibly its transpose C7. In the context of solving linear systems Ax = b, the matrix Ag is called 
a preconditioner, and the matrix C is called the preconditioned coefficient matrix. 

In general, the matrix C = Aj ' A is full. However, if C is only used in the form of matrix-vector 
products, then there is no need to explicitly form C. Instead, for any v € C”, the result of the 
matrix-vector product y = Cv can be computed as follows: 


Set c= Av, 
Set d= Pc, 
Solve Lf=d for f, 
Solve Uz=f for z, 
Set y= Q?z. 
Since P and Q are permutation matrices, the second and the last steps are just reorderings of 


the entries of the vectors c and z, respectively. Therefore, the main computational cost is the 
matrix-vector product with the sparse matrix A in the first step, the triangular solve with L in the 
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third step, and the triangular solve with U in the fourth step, which requires a total of at most 
2(nnz(A) + nnz(L) + nnz(U)) flops. Similarly, each matrix product with C? can be computed 
with at most 2(mnz(A) + nnz(L) + nnz(U)) flops. In particular, matrix-vector products with both 
C and C! are fast. 

4. For sparse Hermitian matrices A = A* € C”*", preconditioning is often applied in a symmetric 
manner. Suppose 


Ao = PLL* PT 


is a sparse Cholesky factorization of a sparse matrix Ay = Aj € C"*", Ap > 0, which in some sense 
approximates the original matrix A. Then the symmetrically preconditioned matrix C is defined 
as 


C := (PL)! A(L*P?)"! = L7! PT AP(L*)“!. 


Note that C = C* is a Hermitian matrix. For any v € C”, the result of the matrix-vector product 
y = Cvcan be computed as follows: 


Solve L*c=v for c¢, 


Set d= Pec, 
Set f= Ad, 
Set z= P'f, 


Solve Ly=z for y. 


The main computational cost is the triangular solve with L* in the first step, the matrix-vector 
product with the sparse matrix A in the third step, and the triangular solve with L in the last step, 
which requires a total of at most 2(nnz(A) + 2nnz(L)) flops. In particular, matrix-vector products 
with C are fast. 


49.3. Krylov Subspaces 


Petrov—Galerkin projections are often used in conjunction with Krylov subspaces. In this section, we 
present the basic concepts of Krylov subspaces. In the following, it is assumed that C € C”*" andr € C", 


r0. 
Definitions: 
The sequence 
r, Cr, C’r, ot Ci y,, so 


is called the Krylov sequence induced by C and r. 
Let j > 1. The subspace 


K;(C,r) := span{ r,Cr,C’r,...,Ci'r} 


of C” spanned by the first j vectors of the Krylov sequence is called the jth Krylov subspace induced by 
C andr. 
A sequence of linearly independent vectors 


V1,V2>--.,Vj €C" 
is said to be a nested basis for the jth Krylov subspace K ;(C,r) if 


span{vj,V2,...,v;} = Ki(C,r) forall i= 1,2,...,7. 
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Let p(A) = co tej A + cod? +--+ ¢g_1A7-! + 44 be a monic polynomial of degree d with coefficients 
in C. The minimal polynomial of C with respect to r is the unique monic polynomial of smallest possible 
degree for which p(C)r = 0. 

The grade of C with respect to r, d(C,r), is the degree of the minimal polynomial of C and r. 


Facts: 
The following facts can be found in [Hou75, Sect. 1.5], [SB02, Sect. 6.3], or [Saa03, Sect. 6.2]. 


1. The vectors 
r, Cr, C’r, meer city 


are linearly independent if and only if j < d(C,r). 
2. Let d = d(C,r). The vectors 


r,Cr,C’r,...,C??r, C4 '4, Cir 


are linearly dependent for all j > d. 
3. The dimension of the jth Krylov subspace K j(C,r) is given by 


j if j < d(C,r), 


dim K;(C,r) = Les if j > d(C,r). 


4. d(C,r) = rank[r Cr Cr Stee c"!y]. 


49.4 The Symmetric Lanczos Process 


In this section, we assume that C = C* € C”*" is a Hermitian matrix and thatr € C”, r ~ 0 isa nonzero 
starting vector. We discuss the symmetric Lanczos process [Lan50] for constructing a nested basis for the 
Krylov subspace K ;(C,r) induced by C and r. 


Algorithm (Symmetric Lanczos process) 
Compute £; = ||r|l2, and set v, = r/f), and vp = 0. 
For j = 1,2,..., do: 


1) Compute v = Cvj, and set v = v — vj-1 fj. 
2) Compute a; = viv and set V = v — vjq;. 
3) Compute B41 = [Ivll2. 

If Bj41 = 0, stop. 

Otherwise, set Vj. = V/Bj41. 


end for 


Facts: 
The following facts can be found in [CW85], [SB02, Sect. 6.5.3], or [Saa03, Sect. 6.6]. 


1. In exact arithmetic, the algorithm stops after a finite number of iterations. More precisely, it stops 
when j = d(C,r) is reached. 
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2: 


10. 


The Lanczos vectors 
V5 Vay +005 Vj 


generated during the first j iterations of the algorithm form a nested basis for the jth Krylov 
subspace Kj(C,r). 
The Lanczos vectors satisfy the three-term recurrence relations 


Vi4i Bin. = Cvi — Vii —Vi-1Bi, 1 =1,2,...,]. 
These three-term recurrence relations can be written in compact matrix form as follows: 


CV, = VjT) + Biavjne; = Vie T,”. 


Here, we set 
Vi =m vw: vl ef =[0 0 --- 0 1lleR*), 
a) Bo 0 0 
B. a2 Bs . 
Tt <«- . . Te!) — Ai and V;,,;=[V; v; 
3, . . . . 
j 0 B 0}? j Bjyset 5) j+l [ J j4t] 
j 
Bj 
0 oe 0 Bj aj 


Note that T; ¢ C/*! and T;° € C'+)*) are tridiagonal matrices. 
In exact arithmetic, the Lanczos vectors are orthonormal. Since the Lanczos vectors are the columns 
of V;, this orthonormality can be stated compactly as follows: 


ViVi =I; and ViVj+i = 0. 


These orthogonality relations, together with the above compact form of the three-term recurrence 
relations, imply that 


T; = ViCV;. 


Thus, the jth Lanczos matrix T; is the orthogonal Petrov—Galerkin projection of C onto the jth 
Krylov subspace K ;(C,r). 

The computational cost of each jth iteration of the symmetric Lanczos process is fixed, and it 
is dominated by the matrix-vector product v = Cv;. In particular, the computational cost for 
generating the orthogonal Petrov—Galerkin projection T; of C is dominated by the j matrix-vector 
products with C. 

If C is a sparse matrix or a preconditioned matrix with a sparse preconditioner, then the matrix— 
vector products with C are fast. In this case, the symmetric Lanczos process is a very efficient 
procedure for computing orthogonal Petrov—Galerkin projections T; of C onto Krylov subspaces 
K j (C > r) . 

The three-term recurrence relations used to generate the Lanczos vectors explicitly enforce orthog- 
onality only among each set of three consecutive vectors, vj_1, vj, and vj41. As a consequence, 
in finite-precision arithmetic, round-off error will usually cause loss of orthogonality among all 
Lanczos vectors Vj, V2,..-5 Vj4i- 

For applications of the Lanczos process in large-scale matrix computations, this loss of orthogonality 
is often benign, and only delays convergence. More precisely, in such applications, the Lanczos 
matrix T; € C/*) for some j < n is used to obtain an approximate solution of a matrix problem 
involving the large matrix C € C”*”. Due to round-off error and the resulting loss of orthogonality, 
the number j of iterations that is needed to obtain a satisfactory approximate solution is larger 
than the number of iterations that would be needed in exact arithmetic. 
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49.5 The Nonsymmetric Lanczos Process 


In this section, we assume that C € C”™” is a general square matrix, and thatr € C",r 40, andle C’, 
14 0, is a pair of right and left nonzero starting vectors. The nonsymmetric Lanczos process [Lan50] is 
an extension of the symmetric Lanczos process that simultaneously constructs a nested basis for the Krylov 
subspace K ;(C,r) induced by C and r, and a nested basis for the Krylov subspace K j(C T 1) induced by 
C? and I. In the context of the nonsymmetric Lanczos process, K j(C,r) is called the jth right Krylov 
subspace, and K j(C T 1) is called the jth left Krylov subspace. 


Algorithm (Nonsymmetric Lanczos process) 
Compute ; = ||rll2, 71 = |[Il]z, and set v) = r/61, w1 = 1/71, Vo = Wo = 0, and dy = 1. 
For j = 1,2,..., do: 


1) Compute 6; = Wi Vj. 
If 5; = 0, stop. 
2) Compute v = Cvj, and set 6; = nj6;/6;-1 and v = v — vj-1;. 
3) Compute aj = wi V, and set v = v— vjqj. 
4) Compute w = C/w;, and set y; = 0;5;/5;-) and w= w— wjaj — wj-17;- 
5) Compute pj 41 = IIvll2 and njs1 = [lwll2. 
If pj41 = 0 or nj41 = O, stop. 
Otherwise, set Vj. = V/j+1 and Wj41 = W/nj41- 


end for 


Facts: 
The following facts can be found in [SB02, Sect. 8.7.3] or [Saa03, Sect. 7.1]. 


1. The occurrence of 6; = 0 in Step 1 of the nonsymmetric Lanczos process is called an exact 
breakdown. In finite-precision arithmetic, one also needs to check for 6; ~* 0, which is called a 
near-breakdown. It is possible to continue the nonsymmetric Lanczos process even if an 
exact breakdown or a near-breakdown has occurred, by using so-called “look-ahead” techniques. 
(See, e.g., [FGN93] and the references given there.) However, in practice, exact breakdowns and 
even near-breakdowns are fairly rare and, therefore, here we consider only the basic form of the 
nonsymmetric Lanczos process without look-ahead. 

2. In exact arithmetic and if no exact breakdowns occur, the algorithm stops after a finite number of 
iterations. More precisely, it stops when j = min{ d(C,r), d(C T 1) } is reached. 

3. The right Lanczos vectors and the left Lanczos vectors 


Vi, V2,--.,Vj and W),W2,...,W;j 


generated during the first j iterations of the algorithm form a nested basis for the jth right Krylov 
subspace K ;(C,r) and the jth left Krylov subspace K j(C TD), respectively. 
4. The right and left Lanczos vectors satisfy the three-term recurrence relations 


Visi Pi41 = CV; — Via; —Vji-1Bi, 1=1,2,..., 4, 
and 
unis = Cw —wiag —wi-iyi, i= 1,2 j 
Wi+i7i+1 = Wi Wi Qj Wi-1Vi> 1=1, peeea do 


respectively. 
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5. These three-term recurrence relations can be written in compact matrix form as follows: 
T (e) 
CVj= VjTj + oj4iVj4ie; = VioiT; » 


CTWi= WiTj + njriwirie;- 


Here, we set 

Viel vw vib Weal we wih 
ay Bo 0 eee 0 ay V2 0 eee 0 
P2 a2 Bs N2 2 3 

Ti=10 3 Oe SE noe ag o|> 
: ‘ Bj : j - Yj 
OO --- 0 Pj a; QO «.. 0 Nj Qj 

r ise} (e) Tj 

e-=[(0 0 :-- 0 I]E€R%, and Te’ = r|- 

y : Pj+1€; 


Note that T;, T; € C/*/, and ie € CYtD*I are tridiagonal matrices. 
6. The matrix no has full rank, i.e., rank ey =j. 
7. In exact arithmetic, the right and left Lanczos vectors are biorthogonal to each other, i.e., 


0 ifi Fk, 
wi, = ‘ forall i,k =1,2,...,j. 
6; ifi =k, 
Since the right and left Lanczos vectors are the columns of V; and Wj, respectively, the biorthogo- 
nality can be stated compactly as follows: 
T 
WIV; =D 


T. i: 
p> W; Viti = 0, and V; Wj = 0. 


Here, D; is the diagonal matrix 
D; = diag(5,, 52, Seis ,6;). 


Note that Dj; is nonsingular, as long as no exact breakdowns occur. 
8. These biorthogonality relations, together with the above compact form of the three-term recurrence 
relations, imply that 


i -1,,-T 
T; = Dj'V}CV; = (WIV) WFCV,. 


Thus, the jth Lanczos matrix T; is the oblique Petrov—Galerkin projection of C onto the jth right 
Krylov subspace K ;(C,r), and orthogonally to the jth left Krylov subspace K ;(C",1). 
9. The matrices T; and Tr are diagonally similar: 


“TT -l 
TT = DIED, 


10. The computational cost of each jth iteration of the nonsymmetric Lanczos process is fixed, and 
it is dominated by the matrix-vector product v = Cv; with C and by the matrix-vector product 
w = C'w; with C’. In particular, the computational cost for generating the oblique Petrov— 
Galerkin projection T; of C is dominated by the j matrix-vector products with C and the j 
matrix-vector products with C’. 
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11 


12. 


13. 


14. 


. If C is a sparse matrix or a preconditioned matrix with a sparse preconditioner, then the matrix- 
vector products with C and C? are fast. In this case, the nonsymmetric Lanczos process is a very 
efficient procedure for computing oblique Petrov-Galerkin projections T; of C onto right Krylov 
subspaces K ;(C,r), and orthogonally to left Krylov subspaces K j(C T 1). 

The three-term recurrence relations, which are used to generate the right and left Lanczos vectors, 
explicitly enforce biorthogonality only between three consecutive right vectors, vj_1, Vj, Vj+1, and 
three consecutive left vectors, wj_1, Wj, Wj+1- AS a consequence, in finite-precision arithmetic, 
round-off error will usually cause loss of biorthogonality between all right vectors v, V2, ...;Vj+1 
and all left vectors w1, W2,...,Wj+1- 

For applications of the Lanczos process in large-scale matrix computations, this loss of orthogonality 
is often benign, and only delays convergence. More precisely, in such applications, the Lanczos 
matrix T; € C/*/ for some j <n is used to obtain an approximate solution of a matrix problem 
involving the large matrix C € C”*”. Due to round-off error and the resulting loss of biortho- 
gonality, the number j of iterations that is needed to obtain a satisfactory approximate solution is 
larger than the number of iterations that would be needed in exact arithmetic. (See, e.g., [CW86].) 
If C = C* is a Hermitian matrix and | = f, ie., the left starting vector | is the complex conjugate 
of the right starting vector r, then the right and left Lanczos vectors satisfy 


w=v; forall i=1,2,...,j +1 


and the nonsymmetric Lanczos process reduces to the symmetric Lanczos process. 


49.6 The Arnoldi Process 


The Arnoldi process [Arn51] is another extension of the symmetric Lanczos process for Hermitian matri- 
ces to general square matrices. Unlike the nonsymmetric Lanczos process, which produces bases for both 


right 


and left Krylov subspaces, the Arnoldi process generates basis vectors only for the right Krylov sub- 


spaces. However, these basis vectors are constructed to be orthonormal, resulting in a numerical procedure 
that is much more robust than the nonsymmetric Lanczos process. 

In this section, we assume that C € C”*" is a general square matrix, and thatr € C”,r 4 0, isa nonzero 
starting vector. 


For 


Algorithm (Arnoldi process) 
Compute (; = ||r||2, and set v, = r/p1. 


j =1,2,...,do: 


1) Compute v = Cv;. 
2) Fori = 1,2,..., 7, do: 
Compute h;; = v*v;, and set v = v—vjh 
end for 


ije 


3) Compute hj+1,; = |lvll2. 
If hj41,j = 0, stop. 
Otherwise, set vj41 = v/hj41,;- 


end for 


Facts: 


The following facts can be found in [Saa03, Sect. 6.3]. 


1 


. In exact arithmetic, the algorithm stops after a finite number of iterations. More precisely, it stops 
when j = d(C,r) is reached. 
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2; 


10. 


The Arnoldi vectors 
V5 V25 +05 Vj 


generated during the first j iterations of the algorithm form a nested basis for the jth Krylov 
subspace K j(C,r). 


. The Arnoldi vectors satisfy the (i + 1)-term recurrence relations 


Vis hii = Cvi — Wihig — Vi-vhi-1i — +++ — Vohoi —vihii, 1 = 1,2,..., 7). 
These (i + 1)-term recurrence relations can be written in compact matrix form as follows: 


T (e) 
CV) = VjHj + hisnivine; = Vivi Hj- 


Here, we set 
Vi=ly w --- vj) ef =[0 0 --- 0 ER”, 
hy hy hy hij 
ho hy has 
Hj=10 hy hj-2,; 
“ hj-u,j 
0 0  hyj-ar ij 


Note that Hj € C/*) and Hy” € C\+)*J are upper Hessenberg matrices. 


. The matrix Hy has full rank, i.e., rank H Bg =j. 
. Since the Arnoldi vectors are the columns of Vj, this orthonormality can be stated compactly as 


follows: 


Viv; = I; and ViVi+1 = 0. 


. These orthogonality relations, together with the above compact form of the recurrence relations, 


imply that 
Hj = V}CVj. 


Thus, the jth Arnoldi matrix H; is the orthogonal Petrov-Galerkin projection of C onto the jth 
Krylov subspace K ;(C,r). 

As in the case of the symmetric Lanczos process, each jth iteration of the Arnoldi process requires 
only a single matrix-vector product v = Cvj. If C is a sparse matrix or a preconditioned matrix 
with a sparse preconditioner, then the matrix-vector products with C are fast. 


. However, unlike the Lanczos process, the additional computations in each jth iteration do increase 


with j. In particular, each jth iteration requires the computation of j inner products of vectors of 
length n, and the computation of j SAXPY-type updates of the form v = v — v,h;; with vectors of 
length n. 

For most large-scale matrix computations, the increasing work per iteration limits the number of 
iterations that the Arnoldi process can be run. Therefore, in practice, the Arnoldi process is usually 
combined with restarting; i.e., after a number of iterations (with the matrix C and starting vector r), 
the algorithm is started again with the same matrix C, but a different starting vector, say rj). 

On the other hand, the (i + 1)-term recurrence relations used to generate the Arnoldi vectors 
explicitly enforce orthogonality among the first i + 1 vectors, v1, V2,...,Vj41. As a result, the 
Arnoldi process is much less susceptible to round-off error in finite-precision arithmetic than the 
Lanczos process. 
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49.7 Eigenvalue Computations 


In this section, we consider the problem of computing a few eigenvalues, and possibly eigenvectors, of a 
large matrix C € C"*". We assume that matrix-vector products with C are fast. In this case, orthogonal 
and, in the non-Hermitian case, oblique Petrov-Galerkin projections of C onto Krylov subspaces K ;(C,r) 
can be computed efficiently, as long as j <n. 


Facts: 
The following facts can be found in [CW85], [CW86], and [BDD00]. 


1. Assume that C = C* € C"*” isa Hermitian matrix. We choose any nonzero starting vectorr € C”, 
r # 0, e.g., a vector with random entries, and run the symmetric Lanczos process. After j iterations 
of the algorithm, we have computed the jth Lanczos matrix T;, which — in exact arithmetic — is 
the orthogonal Petrov—Galerkin projection of C onto the jth Krylov subspace K ;(C, r). Neglecting 
the last term in the compact form of the three-term recurrence relations used in the first j iterations 
of the symmetric Lanczos process, we obtain the approximation 


CV) © VT}. 


This approximation suggests to use the j eigenvalues a epee j, of the jth Lanczos 
matrix T; €¢ C’*/ as approximate eigenvalues of the original matrix C. Furthermore, if one is also 
interested in approximate eigenvectors, then the above approximation suggests to use 


x? = Vj2i” eC", where Tx a Ze, Zi) + 0, 


as an approximate eigenvector of C corresponding to the approximate eigenvalue A GEG: 

2. Assume that C € C"*" is a general square matrix. Here one can use either the nonsymmetric 
Lanczos process or the Arnoldi process to obtain approximate eigenvalues. 

3. In the case of the nonsymmetric Lanczos process, one chooses any nonzero starting vectors r € C”, 
r#0,andleC",140,r € C",r 4 0. In analogy to the symmetric case, the eigenvalues of Lanc- 
zos matrix T; € C/*) computed by j iterations of the nonsymmetric Lanczos process are used as 
approximate eigenvalues of the original matrix C. Corresponding approximate right eigenvectors 
are given by the same formula as above. Furthermore, one can also obtain approximate left eigen- 
vectors from the left eigenvectors of T; and the first j left Lanczos vectors. A discussion of many 
practical aspects of using the nonsymmetric Lanczos process for eigenvalue computations can be 
found in [CW86]. 

4. In the case of the Arnoldi process, one only needs to choose a single nonzero starting vector r € C", 
r 4 0. Here, one has the approximation 


G Vj ~ Vj H pP 
where V; is the matrix containing the first j Arnoldi vectors as columns and Hj is the jth Arnoldi 
matrix. The eigenvalues av B i=1,2,...,j,of Hj € C/*/ are used as approximate eigenvalues of 


C. Furthermore, for each i, 


x!) = Vjzi” €C", where Hy? = ZIP 2) #0, 


i i 


is an approximate eigenvector of C corresponding to the approximate eigenvalue reo SEC. 


49.8 Linear Systems of Equations 


In this section, we consider the problem of solving large systems of linear equations, 
Cx =b, 


where C € C"*" is a nonsingular matrix and b € C”. We assume that any possible preconditioning 
was already applied and so, in general, C is a preconditioned version of the original coefficient matrix. 
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In particular, the matrix C may actually be dense. However, we assume that matrix-vector products with 
C and possibly C7 are fast. This is the case when C is a preconditioned version of a sparse matrix A anda 
preconditioner Ao that allows a sparse LU or Cholesky factorization. 


Facts: 
The following facts can be found in [FGN92] or [Saa03]. 


1. Let x9 € C” be an arbitrary initial guess for the solution of the linear system, and denote by 
fo = b— Cxo 


the corresponding residual vector. A Krylov subspace-based iterative method for the solution of 
the above linear system constructs a sequence of approximate solutions of the form 


Xj € Xo + Kj(C,140), 5 0) by ene 


i.e., the jth iterate is an additive correction of the initial guess, where the correction is chosen from 
the jth Krylov subspace K j(C, ro) induced by the coefficient matrix C and the initial residual ro. 
Now let V; € C"*! be a matrix the columns of which form a nested basis for K j(C, 10). Then, any 
possible jth iterate can be parametrized in the form 


Xj = Xo + VjZj> where Zj E Ci. 
Moreover, the corresponding residual vector is given by 
Yj =b-Cx; = 1o — CVjzZ;. 


Different Krylov subspace-based iterative methods are then obtained by specifying the choice of 
the basis matrix V; and the choice of the parameter vector z;. 

2. The biconjugate gradient algorithm (BCG) [Lan52] employs the nonsymmetric Lanczos process 
to generate nested bases for the right Krylov subspaces K ;(C,ro) and the left Krylov subspaces 
K,(C T 1). Here, 1 € C",14 0, is an arbitrary nonzero starting vector. The biorthogonality of the 
right and left Lanczos vectors is exploited to construct the jth iterate x; such that the corresponding 
residual vector r; is orthogonal to the left Lanczos vectors, i.e., W/r j = 0. Using the recurrence 
relations of the Lanczos process and the above relation for rj, one can show that the defining 
condition Wr j = 0 is equivalent to z; being the solution of the linear system 


(j) 
TjZ; = ey’ Pi» 


where el! denotes the first unit vector of length j. Moreover, the corresponding iterates x; can be 
obtained via a simple update from the previous iterate x;_, resulting in an elegant overall compu- 
tational procedure. Unfortunately, in general, it cannot be guaranteed that all Lanczos matrices T; 
are nonsingular. As a result, BCG iterates x; may not exist for every j. More precisely, BCG breaks 
down if T; is singular, and it exhibits erratic convergence behavior when T; is nearly singular. 

3. The possible breakdowns and the erratic convergence behavior can be avoided by replacing the 
j X j linear system Tz; = ey pr by the (j + 1) x j least-squares problem 

min Je Po — T}4| ; 

zeC’ 2 
Since ie € CY+)*J always has full rank j, the above least-squares problem has a unique solution z j: 
The resulting iterative procedure is the quasi-minimal residual method (QMR) [FN91]. 

4. The generalized minimal residual algorithm (GMRES) [SS86] uses the Arnoldi process to generate 
orthonormal basis vectors for the Krylov subspaces K j(C, ro). The orthonormality of the columns 
of the Arnoldi basis matrix V; allows one to choose z; such that the residual vector r; has the 
smallest possible norm, i.e., 

Ix {ll2 = Ito — CVjzjll2 = min ||ro — CVjzII2. 
zeC’ 
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Using the compact form of the recurrence relations used to generate the Arnoldi vectors, one 
readily verifies that the above minimal residual property is equivalent to z; being the solution of 
the least-squares problem 


(j+1) 


min je" p — Hy2 


zeC’ 


> 


where H ee € C¥+)*) is an upper Hessenberg matrix. 

5. The idea of quasi-minimization of the residual vector can also be applied to Lanczos-type iterations 
that, in each jth step, perform two matrix—vector products with C, instead of one product with 
C and one product with C’. The resulting algorithm is called the transpose-free quasi-minimal 
residual method (TFQMR) [Fre93]. We stress that QMR and TFQMR produce different sequences 
of iterates and, thus, QMR and TFQMR are not mathematically equivalent algorithms. 


49.9 Dimension Reduction of Linear Dynamical Systems 


In this section, we discuss the application of the nonsymmetric Lanczos process to a large-scale matrix 
problem that arises in dimension reduction of time-invariant linear dynamical systems. A more detailed 
description can be found in [Fre03]. 


Definitions: 


Let A, E € C"*". The matrix pencil A — sE, s € C, is said to be regular if the matrix A — sE is singular 
only for finitely many values s € C. 

A single-input, single-output, time-invariant linear dynamical system is a system of differential- 
algebraic equations (DAEs) of the form 


d 
E—x= Ax+ bu(t), 
dt 


y(t) =1"x(0), 


together with suitable initial conditions. Here, A, E € C”*” are given matrices such that A —sE isa regular 
matrix pencil, b € C”, b 4 0, andl € C",1 £ 0, are given nonzero vectors, x(t) € C” is the vector of state 
variables, u(t) € C is the given input function, y(t) € C is the output function, and n is the state-space 
dimension. 

The rational function 


H:C#CUoo, H(s):=1' (sE— A)'b, 


is called the transfer function of the above time-invariant linear dynamical system. 
A reduced-order model of state-space dimension j (< ) of the above system is a single-input, single- 
output, time-invariant linear dynamical system of the form 


E,=2z= AjZ + bju(t), 
y(t) =]; 2(t), 


where Aj, Ej € C/*/ andb pl € Ci, together with suitable initial conditions. 

Let so € C be such that the matrix A — soE is nonsingular. A reduced-order model of state-space 
dimension j of the above system is said to be a Padé model about the expansion point so if the matrices 
Aj, E; and the vectors b;, 1; are chosen such that the Taylor expansions about so of the transfer function 
H of the original system and of the reduced-order transfer function 


Hj :CrCUnm, Hy(s) == 1} (sEj — Aj)“'b;, 
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agree 


in as many leading Taylor coefficients as possible, i-e., 


Hj(s) = H(s) + O((s — 5o)4), 


where q(/j) is as large as possible. 


Facts: 
The following facts can be found in [FF94], [FF95], or [Fre03]. 


1 
2 


3. 


. In the “generic” case, q(j) = 2). 
. In the general case, q(j) > 2j; the case q(j) > 27 occurs only in certain degenerate situations. 
The transfer function H can be rewritten in terms of a single square matrix C € C"*” as follows: 


H(s) = I" (I, + (s = s9)C)~"s, where C := (soE — A)'E, ri= (spE — A) 'b. 


Note that the matrix C can be viewed as a preconditioned version of the matrix E using the 
“shift-and-invert” preconditioner so E — A. 

. In many cases, the state-space dimension n of the original time-invariant linear dynamical system 
is very large, but the large square matrices A, E € C”*” are sparse. Furthermore, these matrices 
are usually such that sparse LU factorizations of the shift-and-invert preconditioner soE — A 
can be computed with limited amounts of fill-in. In this case, matrix-vector products with the 
preconditioned matrix C and its transpose C7 are fast. 

. The above definition of Padé models suggests the computation of these reduced-order models by 
first explicitly generating the leading q (j) Taylor coefficients of H about the expansion point so, and 
then constructing the Padé model from these. However, this process is extremely ill-conditioned 
and numerically unstable. (See the discussion in [FF94] or [FF95].) 

. A much more stable way to compute Padé models without explicitly generating the Taylor coef- 
ficients is based on the nonsymmetric Lanczos process. The procedure is simply as follows: One 
uses the vectors r and 1 from the above representation of the transfer function H as right and left 
starting vectors, and applies the nonsymmetric Lanczos process to the preconditioned matrix C. 
After j iterations, the algorithm has produced the j x j tridiagonal Lanczos matrix T;. The 
reduced-order model defined by 


Aj = so1j = Ij, Ej; = T;, b; = (Ir) el, iF = el) 
is a Padé model of state-space dimension j about the expansion point so. Here, el denotes the 
first unit vector of length j. 

. In the large-scale case, Padé models of state-space dimension j <« n often provide very 
accurate approximations of the original system of state-space dimension n. In particular, this 
is the case for applications in VLSI circuit simulation. (See [FF95],[Fre03], and the references 
given there.) 

. Multiple-input multiple-output time-invariant linear dynamical systems are extensions of the 
above single-input single-output case with the vectors b and | replaced by matrices B € C"*™ 
and L € C"*?, respectively, where m is the number of inputs and p is the number of outputs. 
The approach outlined in this section can be extended to the general multiple-input multiple- 
output case. A suitable extension of the nonsymmetric Lanczos process that can handle multiple 
right and left starting vectors is needed in this case. For a discussion of such a Lanczos-type 
algorithm and its application in dimension reduction of general multiple-input multiple- 
output time-invariant linear dynamical systems, we refer the reader to [Fre03] and the references 
given there. 


49-16 Handbook of Linear Algebra 


References 


[Arn51] W.E. Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue 
problem. Quart. Appl. Math., 9:17-29, 1951. 

[BDD00] Z. Bai, J. Demmel, J. Dongarra, A. Ruhe, and H. van der Vorst, Eds., Templates for the Solution 
of Algebraic Eigenvalue Problems: A Practical Guide. SIAM Publications, Philadelphia, PA, 2000. 

[CW85] J.K. Cullum and R.A. Willoughby. Lanczos Algorithms for Large Symmetric Eigenvalue Computa- 
tions, Vol. 1, Theory. Birkhauser, Basel, Switzerland, 1985. 

[CW86] J.K. Cullum and R.A. Willoughby. A practical procedure for computing eigenvalues of large 
sparse nonsymmetric matrices. In J.K. Cullum and R.A. Willoughby, Eds., Large Scale Eigenvalue 
Problems, pp. 193-240. North-Holland, Amsterdam, The Netherlands, 1986. 

[DER89] I.S. Duff, A.M. Erisman, and J.K. Reid. Direct Methods for Sparse Matrices. Oxford University 
Press, Oxford, U.K., 1989. 

[FF94] P. Feldmann and R.W. Freund. Efficient linear circuit analysis by Padé approximation via the Lanczos 
process. In Proceedings of EURO-DAC ’94 with EURO-VHDL 794, pp. 170-175, Los Alamitos, CA, 
1994, IEEE Computer Society Press. 

[FF95] P. Feldmann and R.W. Freund. Efficient linear circuit analysis by Padé approximation via the 
Lanczos process. IEEE Trans. Comp.-Aid. Des., 14:639-649, 1995. 

[Fre93] R.W. Freund. A transpose-free quasi-minimal residual algorithm for non-Hermitian linear 
systems. SIAM J. Sci. Comp., 14:470—482, 1993. 

[Fre03] R.W. Freund. Model reduction methods based on Krylov subspaces. Acta Numerica, 12:267— 
319, 2003. 

[FGN92] R.W. Freund, G.H. Golub, and N.M. Nachtigal. Iterative solution of linear systems. Acta 
Numerica, 1:57—100, 1992. 

[FGN93] R.W. Freund, M.H. Gutknecht, and N.M. Nachtigal. An implementation of the look-ahead 
Lanczos algorithm for non-Hermitian matrices. SIAM J. Sci. Comp., 14:137-158, 1993. 

[FN91] R.W. Freund and N.M. Nachtigal. QMR: a quasi-minimal residual method for non-Hermitian 
linear systems. Num. Math., 60:315-339, 1991. 

[GMS05] P.E. Gill, W. Murray, and M.A. Saunders. SNOPT: An SQP algorithm for large-scale constrained 
optimization. SIAM Rev., 47:99-131, 2005. 

[Hou75] A.S. Householder. The Theory of Matrices in Numerical Analysis. Dover Publications, New York, 
1975. 

[KR91] N.K. Karmarkar and K.G. Ramakrishnan. Computational results of an interior point algorithm 
for large scale linear programming. Math. Prog., 52:555-586, 1991. 

[Lan50] C. Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential 
and integral operators. J. Res. Nat. Bur. Stand., 45:255-282, 1950. 

[Lan52] C. Lanczos. Solution of systems of linear equations by minimized iterations. J. Res. Nat. Bur. 
Stand., 49:33-53, 1952. 

[LM05] A.N. Langville and C.D. Meyer. A survey of eigenvector methods for web information retrieval. 
SIAM Rev., 47(1):135-161, 2005. 

[NW99] J. Nocedal and S.J. Wright. Numerical Optimization. Springer-Verlag, New York, 1999. 

[Saa03] Y. Saad. Iterative Methods for Sparse Linear Systems. SIAM Publications, Philadelphia, PA, 
2nd ed., 2003. 

[SS86] Y. Saad and M.H. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsym- 
metric linear systems. SIAM J. Sci. Statist. Comp., 7(3):856-869, 1986. 

[SBO2] J. Stoer and R. Bulirsch. Introduction to Numerical Analysis. Springer-Verlag, New York, 3rd ed., 
2002. 

[vdV03] H.A. van der Vorst. Iterative Krylov Methods for Large Linear Systems. Cambridge University 
Press, Cambridge, 2003. 

[Wri97] S.J. Wright. Primal-dual interior-point methods. SIAM Publications, Philadelphia, PA, 1997. 


IV 


Applications 


Applications to Optimization 


50 Linear Programming Leonid N. Vaserstein .......... 0.0 c cece cece eee ees 
51 Semidefinite Programming Henry Wolkowicz............. 00s cece cece eens 


Applications to Probability and Statistics 


52 Random Vectors and Linear Statistical Models Simo Puntanen 
ANG: George: P. Fe SIVA ie sb b588 1k si gece wacp'n's stotdlele taiaid doiacacnie wee Soe eleleleiawedece cages 


53 Multivariate Statistical Analysis Simo Puntanen, George A. F. Seber, 
and: George P. A SIAM okie cs yaad sine beet Seas On ENS ATONE ME Ae oes a Vable sae 


54 MarkovChains Beatrice Meini ......0 0... ccc ccc cece cence cence nen eenes 


Applications to Analysis 
55 Differential Equations and Stability © Volker Mehrmann 

GNA TAUANG SIV Kel so ace 6Cs Sask a Gh Aa hts Map eK Aa ee gene gOS tien hn eee S wales 
56 Dynamical Systems and Linear Algebra —_ Fritz Colonius 

and Wolfgang Kliemann ........ 0. ccc cece cc cc eect e ee cen etn nee e nets tees 
57 Control Theory Peter Benner ......... 0.6 enc nett en eens 
58 Fourier Analysis Kenneth Howell ........... 00... c cece ccc eet enece eet ene e es 


Applications to Physical and Biological Sciences 


59 Linear Algebra and Mathematical Physics Lorenzo Sadun..............0.....065 
60 Linear Algebra in Biomolecular Modeling = Zhijun Wu ................0...0 000 


Applications to Computer Science 


61 Coding Theory Joachim Rosenthal 
and Paul Weiner Soin sche ecdietyytad LON WERE SAR ee eee AUR pede nea ReE DNS 


62 Quantum Computation  Zijian Dido... 66... ccc ete ees 


63 Information Retrieval and Web Search Amy N. Langville 
and Carl D. Meyer iin... css sss ctdas athe cues Op evened erin ad dbadesenaaeasds bee 


64 Signal Processing Michael Stewart ....... 0.0.0... ccc ccc cence eee eens 


Applications to Geometry 
65 Geometry Mark Hunacek ....... 0.0... 0c ccc ccc ccc eee e eee eens 


66 Some Applications of Matrices and Graphs in Euclidean Geometry Miroslav 
Fiedler 


Applications to Algebra 


67 MatrixGroups Peter J. Cameron........ 0 cece cece cece cece teen ene n een enees 


68 Group Representations Randall Holmes 
and T. Y. Tam 


69 Nonassociative Algebras = Murray R. Bremner, Lucia I. Murakami, 
and Ivan P. Shestakov 


70 Lie Algebras Robert Wilson 


Applications to 
Optimization 


50 Linear Programming Leonid N. Vaserstein .......... 0.0 c cece cence nee nee 50-1 
What Is Linear Programming? * Setting Up (Formulating) Linear Programs 
* Standard and Canonical Forms for Linear Programs + Standard Row 
Tableaux * Pivoting * Simplex Method * Geometric Interpretation of 
Phase 2 + Duality + Sensitivity Analysis and Parametric Programming + Matrix 
Games * Linear Approximation * Interior Point Methods 
51 Semidefinite Programming Henry Wolkowicz............. 00sec cece cece eens 51-1 
Introduction + Specific Notation and Preliminary Results » Geometry 
+ Duality and Optimality Conditions + Strong Duality without a Constraint 
Qualification + A Primal-Dual Interior-Point Algorithm + Applications of SDP 


SO 


Linear Programming 


50.1 What Is Linear Programming? ..................... 50-1 
50.2. Setting Up (Formulating) Linear Programs......... 50-3 
50.3. Standard and Canonical Forms 
for Linear Programs ............. 0.00... cece eee 50-7 
50.4 Standard Row Tableaux ................0....000005 50-8 
50:5". » PAVOUINg 62.228 as gine ee bik beet cca canes baat 50-10 
50.6 Simplex Method ................. 0. eee eee eee 50-11 
50.7. Geometric Interpretation of Phase 2 ............... 50-13 
508% “DUALITY fstenesnie se ihnery ikon Haaeiiee sects 50-13 
50.9 Sensitivity Analysis and Parametric Programming .. 50-17 
50.10 Matrix Games........... 0. eee ee 50-18 
50.11 Linear Approximation.......................000088 50-20 
Leonid N. Vaserstein 50.12 Interior Point Methods..................0 cece eens 50-23 
Penn State References: si... chedescadapaints po ret doyeeseyeaee reece needs 50-24 


We freely use the textbook [Vas03]. Additional references, including references to Web sites with software, 
can be found in [Vas03], [Ros00, Sec. 15.1],and INFORMS Resources (http://www.informs.org/Resources/). 


50.1 What Is Linear Programming? 


Definitions: 


Optimization is maximization or minimization of a real-valued function, called the objective function, 
on a set, called the feasible region or the set of feasible solutions. 

The values of function on the set are called feasible values. 

An optimal solution (optimizer) is a feasible solution where the objective function reaches an optimal 
value (optimum), i.e., maximal or minimal value, respectively. 

An optimization problem is infeasible if there are no feasible solutions, i.e., the feasible region is empty. 

It is unbounded if the feasible values are arbitrary large, in the case of a maximization problem, or 
arbitrary small, in the case of a minimization problem. 

A mathematical program is an optimization problem where the feasible region is a subset of R’, a finite 
dimensional real space; i.e., the objective function is a function of one or several real variables. 

A linear form in variables x),... , x, is ¢1X1 + -++-+ CyX,, where c; are given numbers. 

An affine function is a linear form plus a given number. 

The term linear function, which is not used here, means a linear form in some textbooks and an affine 
function in others. 

A linear constraint is one of the following three constraints: f < g, f = g, f > g, where f and g are 
affine functions. In standard form, f isa linear form and g is a given number. 
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A linear program is a mathematical program where the objective function is an affine function and 
the feasible region is given by a finite system of linear constraints, all of them to be satisfied. 


Facts: 


For background reading on the material in this subsection see [Vas03]. 


1. 


Linear constraints with equality signs are known as linear equations. The main tool of simplex 
method, pivot steps, allows us to solve any system of linear equations. 


. In some textbooks, the objective function in a linear program is required to be a linear form. 


Dropping a constant in the objective function does not change optimal solutions, but the optimal 
value changes in the obvious way. 


. Solving an optimization problem usually means finding the optimal value and an optimal solution 


or showing that they do not exist. By comparison, solving a system of linear equations usually 
means finding all solutions. In both cases, in real life we only find approximate solutions. 


. Linear programs with one and two variables can be solved graphically. In the case of one variable x, 


the feasible region has one of the following forms: Empty, a point x = a,a finite intervala < x < b, 
witha < b,arayx > aorx < a, the whole line. The objective function f = cx +d is represented 
by a straight line. Depending on the sign of c and whether we want maximize or minimize f, 
we move in the feasible region to the right or to the left as far as we can in search for an optimal 
solution. 

In the case of two variables, the feasible region is a closed convex set with finitely many vertices 
(corners). Together with the feasible region, we can draw in plane levels of the objective function. 
Unless the objective function is constant, every level (where the function takes a certain value) is a 
straight line. This picture allows us to see whether the program is feasible and bounded. If it is, the 
picture allows us to find a vertex which is optimal. 


. Linear programming is about formulating, collecting data, and solving linear programs, and also 


about analyzing and implementing solutions in real life. 


. Linear programming is an important part of mathematical programming. In its turn, mathematical 


programming is a part of operations research. Systems engineering and management science are 
engineering and business versions of operations research. 


. Every linear program is either infeasible, or unbounded, or has an optimal solution. 


Examples: 


ee is 


Here are 3 linear forms in x, y,z: 2x —3y+5z,x+z,y. 
Here are 3 affine functions of x,y,z: 2x —3y+5z—1,x+z+3,y. 
Here are 3 functions of x, y, z that are not affine: xy, x? + z°, sinz. 
The constraint |x| < 1 is not linear, but it is equivalent to a system of two linear constraints, 
x<1,x>-l. 
(An infeasible linear program) Here is a linear program: 
Maximize x + y subject tox < —1,x > 0. 
It has two variables and two linear constraints. The objective function is a linear form. The 


program is infeasible. 
(An unbounded linear program) Here is a linear program: 
Maximize x + y. 
This program has two variables and no constraints. The objective function is a linear form. The 
program is unbounded. 
Here is a linear program: 
Minimize x + y subject tox > l,y > 2. 
This program has two variables and two linear constraints. The objective function is a linear 
form. The optimal value (the maximum) is 3. An optimal solution is x = 1, y = 2. It is unique. 
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Examples: 

1. Finding the maximum of n given numbers ¢),... , C, does not look like a linear program. However, 
it is equivalent to the following linear program with n variables x; and n + 1 linear constraints: 
CyXy bee + CyX, > max, all x; > 0,x, +--- +x, =1. 

An equivalent linear program with one variable y and n linear constraints is y > min, y > c; 
for all i. 

2. (Diet problem [Vas03, Ex. 2.1]). The general idea is to select a mix of different foods in such a 
way that basic nutritional requirements are satisfied at minimum cost. Our example is drastically 
simplified. 

According to the recommendations of a nutritionist, a person’s daily requirements for protein, 
vitamin A, and calcium are as follows: 50 grams of protein, 4000 IUs (international units) of vitamin 
A, 1000 milligrams of calcium. For illustrative purposes, let us consider a diet consisting only of 
apples (raw, with skin), bananas (raw), carrots (raw), dates (domestic, natural, pitted, chopped), 
and eggs (whole, raw, fresh) and let us, if we can, determine the amount of each food to be consumed 
in order to meet the recommended dietary allowances (RDA) at minimal cost. 


Protein Vit.A Calcium 


Food Unit (g) (IU) (mg) 
Apple 1 medium (138 g) 0.3 73 9.6 
Banana 1 medium (118 g) 1.2 96 7 
Carrot 1 medium (72 g) 0.7 20253 19 
Dates 1 cup (178 g) 3.5 890 57 
Egg 1 medium (44 g) 5.5 279 22 


Since our goal is to meet the RDA with minimal cost, we also need to compile the costs of these 


foods: 
Food Cost (in cents) 
1 apple 10 
1 banana 15 
1 carrot 5 
1 cup dates 60 
1 egg 8 


Using these data, we can now set up a linear program. Let a, b,c, d,e be variables representing the 
quantities of the five foods we are going to use in the diet. The objective function to be minimized 
is the total cost function (in cents), 


C = 10a+ 15b + 5c + 60d + 8e, 


where the coefficients represent cost per unit of the five items under consideration. 
What are the constraints? Obviously, 


a,b,c,d,e > 0. (i) 


These constraints are called nonnegativity constraints. 
Then, to ensure that the minimum daily requirements of protein, vitamin A, and calcium are 
satisfied, it is necessary that 


0.3a + 12b + #0.7¢c + 35d + 5.5¢€ > 50 
73a + 96b + 20253c + 890d + 279e > 4000 (ii) 
96a + 7b + 19¢ + 57d + 22e > 1000, 
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where, for example, in the first constraint, the term 0.3a expresses the number of grams of protein 
in each apple multiplied by the quantity of apples needed in the diet, the second term 1.2b expresses 
the number of grams of protein in each banana multiplied by the quantity of bananas needed in 
the diet, and so forth. 
Thus, we have a linear program with 5 variables and 8 linear constraints. 

(Blending problem [Vas03, Ex. 2.2]). Many coins in different countries are made from cupronickel 
(75% copper, 25% nickel). Suppose that the four available alloys (scrap metals) A, B, C, D to be 
utilized to produce the coin contain the percentages of copper and nickel shown in the following 
table: 


Alloy A B Cc D 
% copper 90 80 70 60 
% nickel 10 20 30 40 
$/lb 1.2 1.4 1.7 1.9 


The cost in dollars per pound of each alloy is given as the last row in the same table. 

Notice that none of the four alloys contains the desired percentages of copper and nickel. Our 
goal is to combine these alloys into a new blend containing the desired percentages of copper and 
nickel for cupronickel while minimizing the cost. This lends itself to a linear program. 

Let a, b, c, d be the amounts of alloys A, B, C, D in pounds to make a pound of the new 
blend. Thus, 


a, b,c, d>0. (i) 
Since the new blend will be composed exclusively from the four alloys, we have 
at+b+ct+d=l. (ii) 
The conditions on the composition of the new blend give 


75 
25. 


9a + 8b + 7c + 6d ik 
(iii) 


la + .2b + 3c + Ad 


For example, the first equality states that 90% of the amount of alloy A, plus 80% of the amount 
of alloy B, plus 70% of the amount of alloy C, plus 60% of the amount of alloy D will give the 
desired 75% of copper in a pound of the new blend. Likewise, the second equality gives the desired 
amount of nickel in the new blend. 

Taking the preceding constraints into account, we minimize the cost function 


C= 1.2a+ 1.4b 4 1.7c + 1.9d. 


In this problem, all the constraints, except (i), are equalities. In fact, there are three linear equa- 
tions and four unknowns. However, the three equations are not independent. For example, the 
sum of the equations in (ii7) gives (ii). Thus, (i7) is redundant. 

In general, a constraint is said to be redundant if it follows from the other constraints of our 
system. Since it contributes no new information regarding the solutions of the linear program, it 
can be dropped from consideration without changing the feasible set. 

(Manufacturing problem [Vas03, Ex. 2.3]). We are now going to state a program in which the 
objective function, a profit function, is to be maximized. A factory produces three products: P1, P2, 
and P3. The unit of measure for each product is the standard-sized boxes into which the product 
is placed. The profit per box of P1, P2, and P3 is $2, $3, and $7, respectively. Denote by x), x2, x3 
the number of boxes of P1, P2, and P3, respectively. So the profit function we want to maximize is 


P= 2x, + 3x. + 7x3. 
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The five resources used are raw materials R1 and R2, labor, working area, and time on a machine. 
There are 1200 lbs of R1 available, 300 lbs of R2, 40 employee-hours of labor, 8000 m? of working 
area, and 8 machine-hours on the machine. 

The amount of each resource needed for a box of each of the products is given in the following 
table (which also includes the aforementioned data): 


Resource Unit Pl P2 P3 Available 


Rl lb 40 20 60 1200 
R2 lb 4 1 6 300 
Labor hour 2 a7 2 40 
Area nm 100 100 800 8000 
Machine hour ol 3 6 8 
Profit $ 2 3 7 —> max 


As we see from this table, to produce a box of P1 we need 40 pounds of R1, 4 pounds of R2, 
0.2 hours of labor, 100 m? of working area, and 0.1 hours on the machine. Also, the amount of 
resources needed to produce a box of P2 and P3 can be deduced from the table. The constraints are 


X15 X2,x3 = 0, (i) 
and 
40x, + 20x. + 60x3 < 1200 (pounds of R1) 
4x, + x2 + 6x3; << 300 (pounds of R2) 
2x, + .7xX. + 2x3 < 40 (hours of labor) (ii) 
100x; + 100x, + 800x3 < 8000 (area in m?) 
wx, + 3x. + .8x3 < 8 (machine). 


5. (Transportation problem [Vas03, Ex. 2.4]). Another concern that manufacturers face daily is trans- 
portation costs for their products. Let us look at the following hypothetical situation and try to set 
it up as a linear program. A manufacturer of widgets has warehouses in Atlanta, Baltimore, and 
Chicago. The warehouse in Atlanta has 50 widgets in stock, the warehouse in Baltimore has 30 
widgets in stock, and the warehouse in Chicago has 50 widgets in stock. There are retail stores in 
Detroit, Eugene, Fairview, Grove City, and Houston. The retail stores in Detroit, Eugene, Fairview, 
Grove City, and Houston need at least 25, 10, 20, 30, 15 widgets, respectively. Obviously, the man- 
ufacturer needs to ship widgets to all five stores from the three warehouses and he wants to do this 
in the cheapest possible way. This presents a perfect backdrop for a linear program to minimize 
shipping cost. To start, we need to know the cost of shipping one widget from each warehouse to 
each retail store. This is given by a shipping cost table. 


1D 2E 3.F 4.G 5.H 
1. Atlanta 55 30 40 50 40 
2. Baltimore 35 30 100 45 60 
3. Chicago 40 60 95 35 30 


Thus, it costs $30 to ship one unit of the product from Baltimore to Eugene (E), $95 from Chicago 
to Fairview (F), and so on. 

In order to set this up as a linear program, we introduce variables that represent the number 
of units of product shipped from each warehouse to each store. We have numbered the ware- 
houses according to their alphabetical order and we have enumerated the stores similarly. Let 
xij, forall 1 < i < 3, 1 < j <5, represent the number of widgets shipped from warehouse #i 
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to store #j. This gives us 15 unknowns. The objective function (the quantity to be minimized) is 
the shipping cost given by 


C= 55x11 + 30x12 + 40x13 + 50x14 + 40x15 
+35x21 + 30x22 is 100x23 + 45x24 + 60X25 
+40x31 + 60x32 + 95x33 + 35x34 + 30x35, 


where 55x); represents the cost of shipping one widget from the warehouse in Atlanta to the retail 
store in Detroit (D) multiplied by the number of widgets that will be shipped, and so forth. 
What are the constraints? First, our 15 variables satisfy the condition that 


xij = 0, forall 1<i<3,1<j <5, (i) 


since shipping a negative amount of widgets makes no sense. Second, since the warehouse #i cannot 
ship more widgets than it has in stock, we get 


Xo + XX. + X53 + X44 + X5 < 50 
X2p + Xn + X39 + Xog + 5 < 30 (ii) 
x3) + X32 + X33 + X34 + X35 < 50. 


Next, working with the amount of widgets that each retail store needs, we obtain the following 
five constraints: 


Xi + xX + x3) 2 25 
X12 + X2 + X32 FS 10 
X13 + X23 + x33 > 20 (iii) 
X14 + Xx + X34 2 30 
x15 + X55 + x35 2S 15. 


The problem is now set up. It is a linear program with 15 variables and 23 linear constraints. 
(Job assignment problem [Vas03, Ex. 2.5]). Suppose that a production manager must assign n 
workers to do n jobs. If every worker could perform each job at the same level of skill and efficiency, 
the job assignments could be issued arbitrarily. However, as we know, this is seldom the case. Thus, 
each of the n workers is evaluated according to the time he or she takes to perform each job. The 
time, given in hours, is expressed as a number greater than or equal to zero. Obviously, the goal 
is to assign workers to jobs in such a way that the total time is as small as possible. In order to set 
up the notation, we let c;; be the time it takes for worker #i to perform job #j. Then the times 
could naturally be written in a table. For example, take n = 3 and let the times be given as in the 
following table: 


a b c 
A 10 70 40 
B 20 60 10 
C 10 20 90 


We can examine all six assignments and find that the minimum value of the total time is 40. So, 
we conclude that the production manager would be wise to assign worker A to job a, worker B to 
job c, and worker C to job b. 

In general, this method of selection is not good. The total number of possible ways of assigning 
jobs is n! = n x (n—1) x (n—2) x --- x 2 x 1. This is an enormous number even for moderate 
n. For n = 70, 


n! = 119785716699698917960727837216890987364589381425 
464258575553628646280095827898453 19680000000000000000. 
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It has been estimated that if a Sun Workstation computer had started solving this problem at the 
time of the Big Bang, by looking at all possible job assignments, then by now it would not yet have 
finished its task. 

Although is not obvious, the job assignment problem (with any number n of workers and jobs) 
can be expressed as a linear program. Namely, we set 


xij =0 or 1 (i) 
depending on whether the worker 7 is assigned to do the job j. The total time is then 
SS ye cjjx;j (to be minimized). (ii) 
ar) 
The condition that every worker i is assigned to exactly one job is 
So apa forall 4 (iii) 
j 


The condition that exactly one worker is assigned to every job j is 


Seal forall j. (iv) 


The constraints (i) are not linear. If we replace them by linear constraints x;,; > 0, then we 
obtain a linear program with n? variables and n? + 2n linear constraints. Mathematically, it is a 
transportation problem with every demand and supply equal 1. As such, it can be solved by the 
simplex method (see below). (When n = 70 it takes seconds.) 

The simplex method for transportation problem does not involve any divisions. Therefore, an 
optimal solution it gives integral values for all components x;,;. Thus, the conditions (i) hold, and 
the simplex method solves the job assignment problem. (Another way to express this is that all 
vertices of the feasible region (iii) to (iv) satisfy (i).) 


50.3 Standard and Canonical Forms for Linear Programs 


Definitions: 


LP in canonical form [Vas03] isc’x-+d — min, x > 0, Ax < b, where xis a column of distinct (decision) 
variables, c7 is a given row, d is a given number, A is a given matrix, and b is a given column. 

LP in standard form [Vas03] is c’x + d — min,x > 0, Ax = b, where x, c,d, A, and b are as in the 
previous paragraph. 

The slack variable for a constraint f < c iss = c — f > 0. The surplus variable for a constraint 
f2ciss=f—c>=0. 


Facts: 


For background reading on the material in this section, see [Vas03]. 


1. Every LP can be written in normal as well as standard form using the following five little tricks 
(normal is a term used by some texts): 


(a) Maximization and minimization problems can be converted to each other. Namely, the prob- 
lems f(x) — min,x € S and — f(x) — max,x € S are equivalent in the sense that they have 
the same optimal solutions and max = —min. 

(b) The equation f = g is equivalent to the system of two inequalities, f > g,g < f. 

(c) The inequality f < g is equivalent to the inequality — f > —g. 
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(d) The inequality f < g is equivalent to f + x = g,x > 0, where x = g — f is anew variable 
called a slack variable. 


(e) A variable x unconstrained in sign can be replaced in our program by two new nonnegative 
variables, x = x’ — x”, where x’, x” > 0. 


2. The same tricks are sufficient for rewriting any linear program in different standard, canonical, 
and normal forms used in different textbooks and software packages. In most cases, all decision 
variables in these forms are assumed to be nonnegative. 


Examples: 


1. Thecanonical formc!x+d — min,x > 0, Ax < bcanbe rewritten in the following standard form: 
c'x+d— min, x > 0, y>0, Ax+y=b. 

2. Thestandardformc’?x+d — min,x > 0, Ax = bcanbe rewritten in the following canonical form: 
c'x+d— min, x > 0, Ax < b, —Ax < —b. 


3. The diet problem (Example 2 in Section 50.2) can be put in canonical form by replacing (ii) with 


—0.3a -— 1.2b — 0.7¢ — 35d — 55e€ > —50 
—73a — 96b — 20253c — 890d — 279e > —-—4000 (ii) 
—96a + 7b — 19¢ — 57d — 22e > —1000. 


4. The blending problem (Example 3 in Section 50.2) is in standard form. 


50.4 Standard Row Tableaux 


Definitions: 
A standard row tableau (of [Vas03]) is 


x? |] 
A bl=u 
cl’ d|—> min, x>0,u>0 ORT) 


with given matrix A, columns b and c, and number d, where all decision variables in x, u are distinct. 
This tableau means the following linear program: 


Ax+b=u>0,x>0,c’x+d— min. 


The basic solution for the standard tableau (SRT) is x = 0,u = b. The corresponding value for the 
objective function is d. 

A standard tableau (SRT) is row feasible ifb > 0, i-e., the basic solution is feasible. Graphically, a feasible 
tableau looks like 


@ 1 
x O1=@ 
*  * |—> min, 


where © stands for nonnegative entries or variables. 
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A standard tableau (SRT) is optimal if b > 0 and c > 0. Graphically, an optimal tableau looks like 


@ 1 
x @O/=@ 
@® * |—> min. 
A badrowina standard tableau is a row of the form [O—] = @, where 9 stands for nonpositive entries, — 


stands for a negative number, and © stands for a nonnegative variable. 
® 


A bad column in a standard tableau is a column of the form | . | , where ® stands for a nonnegative 


variable and nonnegative numbers and — stands for a negative number. 


Facts: 


For background reading on the material in this section see [Vas03]. 


1. Standard row tableaux are used in simplex method; see Section 50.6 below. 
2. The canonical form above can be written in a standard tableau as follows: 


x? | 
—-A blj=u 
ce d|>min,x>0,u>0 


where u = b — Ax > 0. 
3. The standard form above can be transformed into canonical form c'x +d — min,x > 0,—Ax < 
—b, Ax < b, which gives the following standard tableau: 


x! 1 

—-A bfl=u 

A -—bl=v 

cf d |—> min, x>0;u,v>0. 


To get a smaller standard tableau, we can solve the system of linear equations Ax = b. If there 
are no solutions, the linear program is infeasible. Otherwise, we can write the answer in the form 
y = Bz+Di, where the column y contains some variables in x and the column z consists of the 
rest of variables. This gives the standard tableau 


ee sa 
B bj=y 
cd! —> min, y>0,z> 0, 


where ¢’'z + d’ is the objective function c’x + b expressed in the terms of z. 

4, The basic solution of an optimal tableau is optimal. 

5. An optimal tableau allows us to describe all optimal solutions as follows. The variables on top with 
nonzero last entries in the corresponding columns must be zeros. Crossing out these columns and 
the last row, we obtain a system of linear constraints on the remaining variables describing the set 
of all optimal solutions. In particular, the basic solution is the only optimal solution if all entries in 
the c-part are positive. 

6. A bad row shows that the linear program is infeasible. 

7. Abad column in a feasible tableau shows that the linear program is is unbounded. 
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Examples: 
1. The linear programs in Example 1 of Section 50.2 and written in an SRT and in an SCT in Example 
1 of Section 50.8 below. 
2. Consider the blending problem, Example 3 in Section 50.2. We solve the system of linear equations 
for a, b and the objective function f and, hence, obtain the standard tableau 


—2 -3 15 |=b 
0.1 012 15 |= f—-> min. 


The tableau is not optimal and has no bad rows or columns. The associated LP can be solved 


graphically, working in the (c, d)-plane. In Example 1 of Section 50.6, we will solve this LP by the 
simplex method. 


50.5 Pivoting 


Definitions: 


Given a system of linear equations y = Az + b solved for m variables, a pivot step solves it for a subset of 
m variables which differs from y in one variable. 
Variables in y are called basic variables. The variables in z are called nonbasic variables. 


Facts: 


For background reading on the material in this section see [Vas03]. 


1. Thus, a pivot step switches a basic and a nonbasic variable. In other words, one variable leaves the 
basis and another variable enters the basis. 
2. Here is the pivot rule: 


© ay; u y 
a* Bl=u 1/a —B/a |=x 
aa 
y dl=v yla 5—By/a|\=v. 
We switch the two variables x and u. The pivot entry a marked by * must be nonzero, 6 represents 
any entry in the pivot row that is not the pivot entry, y represents any entry in the pivot column 


that is not the pivot entry, and 6 represents any entry outside the pivot row and column. 
3. A way to solve any of linear equations Ax = b is to write it in the row tableau 


and move as many constants from the right margin to the top margin by pivot steps. After this we 
drop the rows that read c = c with a constant c. If one of the rows reads cj = c2 with distinct 
constants Cc), C2, the system is infeasible. The terminal tableau has no constants remaining at the 
right margin. If no rows are left, the answer is 0 = 0 (every x is a solution). If no variables are left 
at the right margin, we obtain the unique solution x = b’. Otherwise, we obtain the answer in the 
form y = Bz +D’ with nonempy disjoint sets of variables y, z. 

This method requires somewhat more computations than the Gauss elimination, but it solves 
the system with parametric b. 
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Examples: 
. x+2y =3 
1. Here is a way to solve the system 
4x+7y=5 
by two pivot steps: 
x y 3) Oy 3 5 


50.6 Simplex Method 


The simplex method is the most common method for solving linear programs. It was suggested by Fourier 
for linear programs arising from linear approximation (see below). Important early contributions to linear 
programming were made by L. Walras and L. Kantorovich. The fortuitous synchronization of the advent 
of the computer and George B. Dantzig’s reinvention of the simplex method in 1947 contributed to the 
explosive development of linear programming with applications to economics, business, industrial engi- 
neering, actuarial sciences, operations research, and game theory. For their work in linear programming, 
P. Samuelson (b. 1915) was awarded the Nobel Prize in Economics in 1970, and L. Kantorovich (1912— 
1986) and T. C. Koopmans (1910-1985) received the Nobel Prize in Economics in 1975. 

Now we give the simplex method in terms of standard tableaux. The method consists of finitely many 
pivot steps, separated in two phases (stages). In Phase 1, we obtain either a feasible tableau or a bad row. In 
Phase 2, we work with feasible tableaux. We obtain either a bad column or an optimal tableau. Following 
is the scheme of the simplex method, where LP stands for the linear program: 


(bad row), 
Phase 1 /| LP is infeasible 


( ) es (bad column), 


tableau 


pivot steps Na Phase 2 / LP is unbounded 
ane oes as 
tableau 
pivot steps \\ 


(a) 
tableau ) * 


Both phases are similar and can be reduced to each other. We start Definitions with Phase 2 and Phase 1 
in detail, dividing a programming loop (including a pivot step) for each phase into four substeps. 


Definitions: 
Phase 2 of simplex method. We start with a feasible tableau (SRT). 


1. Is the tableau optimal? If yes, we write the answer: min = d atx = 0,u= b. 

2. Are there any bad columns? If yes, the linear program is unbounded. 

3. (Choosing a pivot entry) We choose a negative entry in the c-part, say, c; in 
column j. Then we consider all negative entries a;,; above to choose our pivot entry. 
For every aj,; < 0 we compute b;/a;,; where b; is the last entry in the rowi. 
Then we maximize bj /a;,; to obtain our pivot entry q;,;. 

4. Pivot and go to Substep 1. 
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Phase 1 of simplex method. We start with a standard tableau (SRT). 


1. Is the tableau feasible? If yes, go to Phase 2. 

2. Are there bad rows? If yes, the LP is infeasible. 

3. (Choosing a pivot entry) Let the first negative number in the b-part be in the row 7. Consider the 
subtableau consisting of the first i rows with the ith row multiplied by —1, and choose the pivot 
entry as in Substep 3 of Phase 2. 

4. Pivot and go to Substep 1. 


A pivot step (in Phase 1 or 2) is degenerate if the last entry in the pivot row is 0, i.e., the (basic) feasible 
solution stays the same. 

A cycle in the simplex method (Phase 1 or 2) is a finite sequence of pivot steps which starts and ends 
with the same tableau. 


Facts: 


For background reading on the material in this section see [Vas03]. 


1. In Phase 2, the current value of the objective function (the last entry in the last row) either improves 
(decreases) or stays the same (if and only if the pivot step is degenerate). 

2. In Phase 1, the first negative entry in the last column increases or stays the same (if and only if the 
pivot step is degenerate). 

3. Following this method, we either terminate in finitely many pivot steps or, after a while, all our 
steps are degenerate, i.e., the basic solution does not change and we have a cycle. 

4. The basic solutions in a cycle are all the same. 

5. To prevent cycling, we can make a perturbation (small change in the b-part) such that none of the 
entries in this part of the tableau is ever 0 (see [Vas03] for details). 

6. Another approach was suggested by Bland. We can make an ordered list of our variables, and then 
whenever there is a choice we choose the variable highest on the list (see [Vas03] for a reference 
and the proof that this rule prevents cycling). Bland’s rule also turns the simplex method into a 
deterministic algorithm, i.e., it eliminates freedom of choices allowed by simplex method. Given 
an initial tableau and an ordered list of variables, Bland’s rule dictates a unique finite sequence of 
pivot steps resulting in a terminal tableau. 

7. With Bland’s rule, the simplex method (both phases) terminates in at most (oe) — 1 pivot steps 
where m is the number of basis variables and n is the number of nonbasic variables (so the tableau 
has m+ 1 rows and n+ 1 columns). The number (eee) here is an upper bound for the number of all 
standard tableaux that can be obtained from the initial tableau by pivot steps, up to permutation of 
the variables on the top (between themselves) and permutation of the variables at the right margin 
(between themselves). 

8. If all data for an LP are rational numbers, then all entries of all standard tableaux are rational 
numbers. In particular, all basic solutions are rational. If the LP is feasible and bounded, then there 
is an optimal solution in rational numbers. However, like for a system of linear equations, the 
numerators and denominators could be so large that finding them could be impractical. 


Examples: 
1. Consider the blending problem, Example 3 in Section 50.2. 
We start with the standard tableau in Example 2 of Section 50.3. The simplex method allows two 
choices for the first pivot entry, namely, 1 or 2 in the first row. We pick 1 as the pivot entry: 


c Ud 1 a d 1 

1* 2 -05};=a 1 —2 0.5 }=c 

—2 -3 15 |=b r> | —2 1 0.5 |=b 

0.1 01 15 |= f—> min 0.1 —0.1 1.55|= f > min. 
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Now the tableau is feasible, and we can proceed with Phase 2: 


a d 1 a c 1 

1 —2* O05 ;/=c 05 —-0.5 0.25 |=d 
—2 1 0.5 |=b tr }|-1.5 —-0.5 0.75 |=b 2 
0.1 —0.1 1.55}= f > min 0.15 0.05 1.525/= f > min 


The tableau is optimal, so min = 1.525 ata = 0,b = 0.75,c = 0, andd = 0.25. 


50.7 Geometric Interpretation of Phase 2 


Definitions: 


A subset S of RN is closed if S contains the limit of any convergent sequence in S. 

A convex linear combination or mixture of two points x,y, is wx + (1 —a)y, whereO <a <1.A 
particular case of a mixture is the halfsum x/2 + y/2. 

The line segment connecting distinct points x, y consists of all mixtures of x, y. 

A set S is convex if it contains all mixtures of its points. 

An extreme point in a convex set is a point that is not the halfsum of any two distinct points in the set, 
ie., the set stays convex after removing the point. 

In the case ofa closed convex set with finitely many extreme points, the extreme points are called vertices. 

Two extreme points in a convex set are adjacent if the set stays convex after deleting the line segment 
connecting the points. 


Facts: 


For background reading on the material in this section see [Vas03]. 


1. The feasible region for any linear program is a closed convex set with finitely many extreme points 
(vertices). 

2. Consider a linear program and the associated feasible tableaux. The vertices of the feasible region 
are the basic solutions of the feasible tableaux. Permutation of rows or columns does not change 
the basic solution. 

3. A degenerate pivot step does not change the basic solution. Any nondegenerate pivot step takes us 
from a vertex to an adjacent vertex with a better value for the objective function. 

4. The set of optimal solutions for any linear program is a closed convex set with finitely many extreme 
points. 

5. The number of pivot steps in Phase 2 without cycles (say, with Bland’s rule) is less than the number 
of vertices in the feasible region. 

6. For (SRT) with m-+ 1 rows and n + 1 columns, the number of vertices in the feasible region is at 

most (ee When n = 1, this upper bound can be improved to 2. When n = 2, this upper bound 
can be improved to m + 2. It is unknown (in 2005) whether, for arbitrary m, n, a bound exists that 
is a polynomial in m + n. 

7. The total number of pivot steps in both phases (with Bland’s rule) is less than ey: 


50.8 Duality 


Definitions: 


A standard column tableau has the form 


(SCT) 
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The associated LP is —y’b + d > max, -—y’A+c? =v’ >0,y>0. 

The basic solution associated with (SCT) is y = 0, v= c. 

The tableau (SCT) is column feasible if c > 0, i.e., the basic solution is feasible. (So, a tableau is optimal 
if and only if it is both row and column feasible.) 

The linear program in (SCT) is dual to that in (SRT). We can write both as the row and the column 
problem with the same matrix: 


LT 


x 1 
~y A b lau x>0,u>0 
| cf d jae y>0,v>0 (ST) 
=v =w _, max 


Facts: 


For background reading on the material in this section, see [Vas03]. 


1. The linear program in the standard row tableau (SRT) can be written in the following standard 
column tableau: 


as —Al 
A bt | x>0,u>0 
ll 1 
uu’ max 


Then the dual problem becomes the row problem with the same matrix. This shows that the dual 
of the dual is the primal program. 
2. The pivot rule for column and row tableaux is the same inside tableaux: 


—x ie | —u I/a  —B/a 
7 ~y | v/a 8—-By/a | 
| I I 


uv x Vv 


3. Ifa linear program has an optimal solution, then the simplex method produces an optimal tableau. 
This tableau is also optimal for the dual program, hence both programs have the same optimal 
values (the duality theorem). 

4. The duality theorem is a deep fact with several interpretations and applications. Geometrically, 
the duality theorem means that certain convex sets can be separated from points outside them 
by hyperplanes. We will see another interpretation of the duality theorem in the theory of matrix 
games (see below). For problems in economics, the dual problems and the duality theorem also 
have important economic interpretations (see examples below). 

5. Ifa linear program is unbounded, then (after writing it in a standard row tableau) the simplex 
method produces a row feasible tableau with a bad column. The bad column shows that the dual 
problem is infeasible. 

6. There is a standard tableau that has both a bad row and a bad column, hence there is an infeasible 
linear program such that the dual program is also infeasible. 

7. Here is another way to express the duality theorem: Given a feasible solution for a linear program 
and a feasible solution for the dual problem, they are both optimal if and only if the feasible values 
are the same. 

8. Given a feasible solution for a linear program and a feasible solution for the dual problem, they 
are both optimal if and only if for every pair of dual variables at least one value is 0, ie., the 
complementary slackness condition v' x + y’u = 0 in terms of (ST) holds. 

9. More precisely, given a feasible solution (x, u) for the row program in (ST) and a feasible solution 
(y, v) for the column program, the difference (c’x+d)— (-yTu + d) between the corresponding 
feasible values is v?x + y7u. 
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10. All pairs (x, u), (y, v) of optimal solutions for the row and column programs in (ST) are described 
by the following system of linear constraints: Ax + b = u > 0,x > 0, -y'A t+ce=v>0y> 
0,v?x + y’u = 0. This is a way to show that solving a linear program can be reduced to finding a 
feasible solution for a finite system of linear constraints. 


Examples: 


(Generalizations of Examples 1 to 4 in section 50.2 above and their duals): 


1. The linear programs cyx; ++ +++¢,X, — max, all x; > 0,x;+---+x, = l,andy > min, y > ¢; 
for alli in Example 1 are dual to each other. To see this, we use the standard tricks: y = y’ — y” 


with y’, y” > 03x; +--- +x, = 1 is equivalent to x} +---+%, <1,-x,) —----—x, <1. 
We obtain the following standard tableau: 


y’ y" 1 
—x J -J -c =® 
1 1 =1 0 =y— min 
— @ — @ — > max, 
where J is the column of n ones, ¢ = [c1,... ,¢n]’, and x = [x),... ,Xn]’. 


2. Consider the general diet problem (a generalization of Example 2): 
Ax >b,x>0, C=c!’x > min, 


where m variables in x represent different foods and n constraints in the system Ax > b represent in- 
gredients. We want to satisfy given requirements b in ingredients using given foods at minimal cost C. 
On the other hand, we consider a warehouse that sells the ingredients at prices y1,... ; Yn > 0. 


Its objective is to maximize the profit P = y’b, matching the price for each food: y’ A < c’. 


We can write both problems in a standard tableau using slack variables u = Ax — b > 0 and 
vi =cl—y'A> 0: 


x! 1 
=y A —b =u x>0,u>0 
1 cP 0 lee ceats y>0,v>0 
=v =P => max. 


So, these two problems are dual to each other. In particular, the simplex method solves both 
problems and if both problems are feasible, then min(C) = max(P). 
The optimal prices for the ingredients in the dual problem are also called dual or shadow prices. 
These prices tell us how the optimal value reacts to small changes in b; see Section 50.9 below. 
3. Consider the general mixing problem (a generalization of Example 3): 


Ax = b,x > 0, C =c!x > min, 


where m variables in x represent different alloys and n constraints in Ax = b represent elements. 
We want to satisfy given requirements b in elements using given alloys at minimal cost C. 

On the other hand, consider a dealer who buys and sells the elements at prices y1,... 5 Yn. A 
positive price means that the dealer sells, and negative price means that the dealer buys. Dealer’s 
objective is to maximize the profit P = yb matching the price for each alloy: y7 A < c. 

To write the problems in standard tableaux, we use the standard tricks and artificial variables: 


u = Ax—b>0,u"’ = —Ax+b=>0; 
vi=c’-yA>0;y=y-y, y¥=0,y'>0. 
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Now we manage to write both problems in the same standard tableau: 


x! 1 
-y [a -b]_. 
-y’ _-A b aaa x > 0; u,u’>0 
=u 
T if : 
1 c 0 =C-—> min yoy =>0;v>0 
=v!’ =P —+ max. 


4. Consider a generalization of the manufacturing problem in Example 4: 
P =c!x—> max, Ax < b,x > 0, 


where the variables in x are the amounts of products, P is the profit (or revenue) you want to 
maximize, constraints Ax < b correspond to resources (e.g., labor of different types, clean water 
you use, pollutants you emit, scarce raw materials), and the given column b consists of amounts of 
resources you have. Then the dual problem 


y'b — min, yA >cl, y>0 


admits the following interpretation. Your competitor, Bob, offers to buy you out at the following 
terms: You go out of business and he buys all resources you have at price y? > 0, matching your 
profit for every product you may want to produce, and he wants to minimize his cost. 

Again Bob’s optimal prices are your resource shadow prices by the duality theorem. The shadow 
price for a resource shows the increase in your profit per unit increase in the quantity bo of the 
resource available or decrease in the profit when the limit b, decreases by one unit. While changing 
bo, we do not change the limits for the other resources and any other data for our program. There 
are only finitely many values of bo for which the downward and upward shadow prices are different. 
One of these values could be the borderline between the values of bo for which the corresponding 
constraint is binding or nonbinding (in the sense that dropping of this constraint does not change 
the optimal value). 

The shadow price of a resource cannot increase when supply bo of this resource increases (the 
law of diminishing returns, see the next section). 

5. (General transportation problem and its dual). We have m warehouses and n retail stores. The 
warehouse #i has a; widgets available and the store #j needs b; widgets. 

It is assumed that the following balance condition holds: 


n 


m 
> qi = bj. 
i=l 


j=l 


If total supply is greater than total demand, the problem can be reduced to the one with the 
balance condition by introducing a fictitious store where the surplus can be moved at zero cost. If 
total supply is less than total demand, the program is infeasible. 

The cost of shipping a widget from warehouse #i to store #] is denoted by cj; and the number of 
widgets shipped from warehouse #i to store #] is denoted by x;;. The linear program can be stated 
as follows: 


m n 
minimize C(x,...5;Xmn) = S S Ci Xij 
i=1 j=l 


n 
subject to Pee j=l,...,m 


i=l 
m 
> Xij < aj, a eee 7) 
j=l 


xij 2 0, i=1l,...,n, PS Ty. Sh 
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The dual program is 


m n 
_ S ajuj + s bjvj > max, wij =cjj tuj—vj =O forall i,j;u>0,v>0. 
i=l j=l 


So, what is a possible meaning of the dual problem? The control variables u;, v; of the dual problem 
are called potentials or “zones.” While the potentials correspond to the constraints on each retail 
store and each warehouse (or to the corresponding slack variables), there are other variables w;; in 
the dual problem that correspond to the decision variable x;; of the primal problem. 

Imagine that you want to be a mover and suggest a simplified system of tariffs. Instead of mn 
numbers c;,;, we use m + n “zones.” Namely, you assign a “zone” uj => 0i = 1,2 to each of the 
warehouses and a “zone” vj; > 0 j = 1,2 to each of the retail stores. The price you charge is 
v; — uj instead of c;,;. To beat competition, you want vj — uj; < cj,; for all i, j. Your profit is 
— So aju; + ¥¢ jv; and you want to maximize it. 

The simplex method for any transportation problem can be implemented using m by n tables 
rather than mn + 1 by m+n + 1 tableaux. Since all pivot entries are +1, no division is used. In 
particular, if all a;, b; are integers, we obtain an optimal solution with integral x;,;. 

Phase 1 is especially simple. If m,n > 2, we choose any position and write down the maximal 
possible number (namely, the minimum of supply and demand). Then we cross out the row or 
column and adjust the demand or supply respectively. If m = 1 < n, we cross out the column. If 
n= 1 < _m,wecross out the row. If m = n = 1, then we cross out both the row and column. Thus, 
we find a feasible solution in m + n — 1 steps, which correspond to pivot steps, and the m+n — 1 
selected positions correspond to the basic variables. 

Every transportation problem with the balance condition has an optimal solution. (See [Vas03] 
for details.) 


50.9 Sensitivity Analysis and Parametric Programming 


Sensitivity analysis is concerned with how small changes in data affect the optimal value and optimal 
solutions, while large changes are studied in parametric programming. 

Consider the linear program given by (SRT) with the last column being an affine function of a parameter 
t, i.e., we replace b, d by affine functions b + b)t, d+ dt ofa parameter t. Then the optimal value becomes 
a function f(t) of t. 


Facts: 


For background reading on the material in this section, see [Vas03]. 


i 


The function f(t) is defined on a closed convex set S on the line, ie., S is one of the following 
“qntervals”: empty set, a point, an intervala < t < b witha < b,arayt > a,arayt < a, the 
whole line. 

The function f(t) is piece-wise linear, i.e., the interval S is a finite union of subintervals S, with 
f(t) being an affine function on each subinterval. 


. The function f(t) is convex in the sense that the set of points in the plane below the plot is a 


convex set. In particular, the function f(t) is continuous. 

In parametric programming, there are methods for computing f(t). The set S is covered by a finite 
set of tableaux optimal for various values of t. The number of these tableaux is at most Coe ye 
Suppose that the LP with t = 0 (ie., the LP given by (SRT)) has an optimal tableau Tp. Let x = 
xu = u be the corresponding optimal solution (the basic solution), and let y = y®,v = v 
be the corresponding optimal solution for the dual problem (see (ST) for notation). Assume that 
the b-part of To has no zero entries. Then the function f(t) is affine in an interval containing 0. 
Its slope, i.e., derivative f’(0) att = 0, is f’(0) = d, + b, 7 v. Thus, we can easily compute f’(0) 
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from Tp. In other words, the optimal tableaus give the partial derivatives of the optimal value with 
respect to the components of given b and d. 

6. If we pivot the tableau with a parameter, the last column stays an affine function of the parameter 
and the rest of the tableau stays independent of parameter. 

7. Similar facts are true if we introduce a parameter into the last row rather than into the last column. 

8. If both the last row and the last column are affine functions of parameter f, then after pivot steps, 
the A-part stays independent of t, the b-part and c-part stay affine functions of t, but the d-part 
becomes a quadratic polynomial in t. So the optimal value is a piece-wise quadratic function of t. 

9. If we want to maximize, say, profit (rather than minimize, say, cost), then the optimal value is 
concave (convex upward), i.e., the set of points below the graph is convex. The fact that the slope 
is nonincreasing is referred to as the law of diminishing returns. 


50.10 Matrix Games 


Matrix games are very closely related with linear programming. 


Definitions: 


A matrix game is given by a matrix A, the payoff matrix, of real numbers. 

There are two players. The players could be humans, teams, computers, or animals. We call them He and 
She. He chooses a row, and She chooses a column. The corresponding entry in the matrix represents what 
she pays to him (the payoff of the row player). Games like chess, football, and blackjack can be thought of 
as (very large) matrix games. Every row represents a strategy, i.e., his decision what to do in every possible 
situation. Similarly, every column corresponds to a strategy for her. The rows are his (pure) strategies and 
columns are her (pure) strategies. 

A mixed strategy is a mixture of pure strategies. In other words, a mixed strategy for him is a probability 
distribution on the rows and a mixed strategy for her is a probability distribution on the columns. 

We write his mixed strategy as columns p = (p;) with p > 0, 5° p; = 1. We write her mixed strategy 
as rows q’ = (qj) with q > 0,0 qj =1. 

The corresponding payoff is p? Aq, the mathematical expectation. 

A pair of strategies (his strategy, her strategy) is an equilibrium or a saddle point if neither player can 
gain by changing his or her strategy. In other words, no player can do better by a unilateral change. (The 
last sentence can be used to define the term “equilibrium” in any game, while “saddle point” is used only 
for zero-sum two-player games; sometimes it is restricted to the pure joint strategies.) 

In other words, at an equilibrium, the payoff p? Aq has maximum as function of p and minimum as 
function of q. 

His mixed strategy p is optimal if his worst case payoff min(p? A) is maximal. The maximum is the 
value of the game (for him). Her mixed strategy q’ is optimal if her worst case payoff min(— Aq) is 
maximal. The maximum is the value of the game for her. 

If the payoff matrix is skew-symmetric, the matrix game is called symmetric. 


Facts: 


For background reading on the material in this section, see [Vas03]. 


1. A pair (i, 7) of pure strategies is a a saddle point if and only if the entry a;,; of the payoff matrix 
A = aj, is both largest in its column and the smallest in its row. 

2. Every matrix game has an equilibrium. 

. A pair (his strategy, her strategy) is an equilibrium if and only if both strategies are optimal. 

4. His payoff p? Aq at any equilibrium (p, q) equals his value of the game and equals the negative of 
her value of the game (the minimax theorem). 

5. To solve a matrix game means to find an equilibrium and the value of the game. 


iss) 
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6. Solving a matrix game can be reduced to solving the following dual pair of linear programs written 
in a standard tableau: 


qo wh oa 
Spica” ib. cade. 3 =~ > 0 
—j! 1 0 —l1 =x*x>0 
a.” | -17 0 =*«>0 
1 0 1 -—-l 0 => min 
I I I I 
* * * X4—> max. 


Here, A is the m by n payoff matrix, 17 is the row of n ones, 1,, is the column of m ones, p is 
his mixed strategy, q? is her mixed strategy. Note that her problem is the row problem and his 
problem is the column problem. Their problems are dual to each other. Since both problems have 
feasible solutions (take, for example, p = 1,,/m,q = 1,,/1), the duality theorem says that min( f) 
= max(g). That is, 

his value of the game = max(A) = min(jz) = —her value of the game. 


Thus, the minimax theorem follows from the duality theorem. 


7. We can save two rows and two columns and get a row feasible tableau as follows: 


q/(u +c) 1 
—p/(A+c) [ ~A-clmI7 Lin =*«>0 
! a1, 0 = -1/(u+c) > min 
I I 
* —1/(A+c¢) —> max. 


Here we made sure that the value of game is positive by adding a number c to all entries of A, i.e., 
replacing A by A+ clin1}. E.g.,c = 1 — min(A). Again his and her problems are dual to each 
other, since they share the same standard tableau. Since the tableau is feasible, we bypass Phase 1. 

8. An arbitrary dual pair (ST) of linear programs can reduced to a symmetric matrix game, with the 
following payoff matrix 


0 —-A —b 
M=/A™ 0 -c 
b! oct 0 


Its size is (m+n-+1) x (m+n-+1), where m x n is the size of the matrix A. 

9. The definition of equilibria makes sense for any game (not only for two-player zero-sum games). 
However, finding equilibria is not the same as solving the game when there are equilibria with 
different payoffs or when cooperation between players is possible and makes sense. 

10. For any symmetric game, the value is 0, and the optimal strategies for the row and column players 
are the transposes of each other. 


Examples: 
1. [Vas03, Ex. 19.3] Solve the matrix game 


5 0 6 1 —2 

2 1 2 1 2 

aS —9 On 5" 2-9 
-9 -8 0 4 2 
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We mark the maximal entries in every column by *. Then we mark the minimal entries in each row 
by ’. The positions marked by both * and’ are exactly the saddle points: 


In this example, the position (i, j) = (2,2) is the only saddle point. The corresponding payoff (the 
value of the game) is 1. 

2. This small matrix game is known as Heads and Tails or Matching Pennies. We will call the players 
He and She. He chooses: heads ( H) or tails (T). Independently, she chooses: H or T. If they choose 
the same, he pays her a penny. Otherwise, she pays him a penny. Here is his payoff in cents: 


H T 
H|-1 1 
He ‘i 1 aay 
There is no equilibrium in pure strategies. The only equilibrium in mixed strategies is ({1/2, 1/2]’, 
[1/2, 1/2]). The value of game is 0. The game is not symmetric in the usual sense. However the 
game is symmetric in the following sense: if we switch the players and also switch H and T for a 
player, then we get the same game. 

3. Another game is Rock, Scissors, Paper. In this game two players simultaneously choose Rock, Scissors, 
or Paper, usually by a show of hand signals on the count of three, and the payoff function is defined 
by the rules Rock breaks Scissors, Scissors cuts Paper, Paper covers Rock, and every strategy ties against 


itself. Valuing a win at 1, a tie at 0, and a loss at —1, we can represent the game with the following 
matrix, where, for both players, strategy 1 is Rock, strategy 2 is Scissors, and strategy 3 is Paper: 


The only optimal strategy for the column player is q’ = [1/3, 1/3, 1/3]. Since the game is sym- 
metric, the value is 0, and q is the only optimal strategy for the row player. 


50.11 Linear Approximation 


Definitions: 


An /?-best linear approximation (fit) of a given column w with n entries by the columns of a given m by 
n matrix A is Ax, where x is an optimal solution for ||w — Ax||, — min. (See Chapter 37 for information 
about || - |lp.) 

In other words, we want the vector w — Ax of residuals (offsets, errors) to be smallest in a certain sense. 


Facts: 
For background reading on the material in this section, see [Vas03]. 


1. Most common values for p are 2, 1,00. In statistics, usually p = 2 and the first column of the 
matrix A is the column of ones. In simple regression analysis, n = 2. In multiple regression, n > 3. 
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In time series analysis, the second column of A is an arithmetic progression representing time; 
typically, this column is [1, 2, ...,™] T (See Chapter 52 for more information.) 

2. If = 1 and the matrix A is a column of ones, we want to approximate given numbers w; by one 
number Y. When p = 2, the best fit is the arithmetic mean ()> w;)/m. When p = 1, the best fits 
are the medians. When p = oo, the best fit is the midrange (min(w;) + max(w;))/2. 

3. When p = 2, the /?-norm is the usual Euclidean norm, the most common way to measure the size 
of a vector, and this norm is used in Euclidean geometry. The best fit is known as the least squares 
fit. To find it, we drop a perpendicular from w onto the column space of A. In other words, we 
want the vector w — Ax to be orthogonal to all columns of A; that is, A™(w— Ax) = 0. This gives 
a system of n linear equations A? Ax = A’w for n unknowns in the column x. The system always 
has a solution. Moreover, the best fit Ax is the same for all solutions x. In the case when w belongs 
to the column space, the best fit is w (this is true for all p). Otherwise, x is unique. (See Section 5.8 
or Chapter 39 for more information about least squares methods.) 

4. The best /°°-fit is also known as the least-absolute-deviation fit and the Chebyshev approximation. 

5. When p = 1, finding the best fit can be reduced to a linear program. Namely, we reduce the 
optimization problem with the objective function |le||; — min, where e = (e;) = w— Ax, toa 
linear program using m additional variables u; such that |e;| < u; for alli. We obtain the following 
linear program with m + n variables a;, u; and 2m linear constraints: 


S uj > min, —u; < wi —A;X <u; for i=1,...,m, 


where A; is the ith row of the given matrix A. 

6. When p = ox, finding the best fit can also be reduced to a linear program. Namely reduce the 
optimization problem with the objective function ||e||,. — min, where e = (e;) = w— Ax, toa 
linear program using an additional variable u such that e;| < u for all. A similar trick was used 
when we reduced solving matrix games to linear programming. We obtain the following linear 
program with n + 1 variables a;, u and 2m linear constraints: 


t—> min, -—u <w;—A;jx<u for i=1,...,m, 


where A; is the ith row of the given matrix A. 
7. Any linear program can be reduced to finding a best /°°-fit. 


Examples: 
1. [Vas03, Prob. 22.7] Find the best /?-fit w = ch? for p = 1,2, 00 given the following data: 


i 1 2 3 
Height h in m 16 15 17 
Weightwinkg 65 60 70 


Compare the optimal values for c with those for the best fits of the form w/h? = c with the same 
p (the number w/h? in kg/m? is known as BMI, the body mass index). Compare the minimums 
with those for the best fits of the form w = b with the same p. 


Solution 


Case p = 1. We could convert this problem to a linear program with four variables and then 
solve it by the simplex method (see Fact 6 above). But we can just solve graphically the nonlinear 
program with the objective function 


f(c) = |65 — 1.6*c| + |60 — 1.5°c| + |70 — 1.77¢| 
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to be minimized and no constraints. The function f(c) is piece-wise affine and convex. The 
optimal solution is c = x; = 65/1.6* * 25.39. This c equals the median of the three observed 
BMIs 65/1.6, 60/1.57 and 70/1.7*. The optimal value is 


min = |65 — ¢,1.67| + |60 — cy 1.57| + |70 — c)1.77| = 6.25. 


To compare this with the best /'-fit for the model w = b, we compute the median b = x; = 65 
and the corresponding optimal values: 


min = |65 — x;| + |60 — x,;| + |70 — x,| = 10. 


So the model w = ch? is better than w = b for our data with the /!-approach. 

Case p = 2. Our optimization problem can be reduced to solving a linear equation for c 
(see Fact 3 above). Also we can solve the problem using calculus, taking advantage of the fact 
that our objective function 


f(c) = (65 — 1.62c)? + (60 — 1.52c)? + (70 — 1.7°c)* 


is differentiable. The optimal solution is c = x. = 2518500/99841 *% 25.225. This x, is not the 
mean of the observed BMIs, which is about 25.426. The optimal value is min ~ 18. 

The mean of w; is 65, and the corresponding minimal value is 5” + 0? + 5? = 50. So, again the 
model w = ch? is better than w = b. 
Case p = oo. We could reduce this problem to a linear program with two variables and then solve 
it by graphical method or simplex method (see Fact 7 above). But we can do a graphical method 
with one variable. The objective function to minimize now is 


f(c) = max(|65 — 1.67c|, |60 — 1.57c|,|70 — 1.7°c|). 
This objective function f (c) is piecewise affine and convex. The optimal solution is 
Coo = 6500/257 © 25.29. 


It differs from the midrange of the BMIs, which is about 25.44. The optimal value is ~ 3. 
On the other hand, the midrange of the weights w; is 65, which gives min = 5 for the model 
w = b with the best /°-fit. So, again the model w = ch? is better than w = b. 
2. [Vas03, Ex. 2, p. 255] A student is interested in the number w of integer points [x, y] in the disc 
x? + y* <r? of radius r. He computed w for some r: 


fr o[ Lh 2 8 4° -5 -6 7 8 ~ 9 
w | 5 13 29 45 81 113 149 197 253 


The student wants to approximate w by a simple formula w = ar + b with constants a, b. But you 
feel that the area of the disc, xr? would be a better approximation and, hence, the best !?-fit of the 
form w = ar* should work even better for the numbers above. 

Compute the best 1?-fit (the least squares fit) for both models, w = ar + b andw = ar’, and 
find which is better. Also compare both optimal values with 


9 
Swi —qi*). 
i=l 


Solution 


For our data, the equation w = ar + b is the system of linear equations Ax = w, where 
w=(5, 13, 29, 45, 81, 113, 149, 197, 253] E. 


T 
PeDisse we Be ee ORD a 
Nac ae Pt oe ed | andx = [f]. 
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The least-squares solution is the solution to A! Ax = Alw,ie., 
a — 
4| = 


9 
So (w; — 4631/15 + 56)” = 48284/15 © 3218.93. 
i=l 


45 9 


285 45 
885 


a 


So, a = 463/15, b = —56. The error 


For w = ar’, the system is Ba = w, where w is as above and B = [1?, 32, 3’, 4, 5’, 6, 77, 87,97] 7. 
The equation B’ Ba = B'w is 415398a = 47598, hence a = 23799/7699 ~ 3.09118. The error 
9 
S| (w; — 2379917 /7699)” = 2117089/7699 ~ 274.982. 
i=l 
So, the model w = ar? gives a much better least-squares fit than the model w = ar + b although 


it has one parameter instead of two. 
The model w = zr? without parameters gives the error 


9 
So (w; — wi)’ = 147409 — 95196m + 1539877 ~ 314.114. 
i=l 


It is known that w;/h? — a asi — oo. Moreover, |w; — mr? |/1; is bounded asi — oo. It follows 
that a — zm for the the best L ,-fitw = ar? for any p > 1 as we use data forr = 1,2,...n with 
n> ©. 


50.12 Interior Point Methods 


Definitions: 


A point x in a convex subset X C R" is interior if 
{x + (x — y)3/lly — xll2:ye X y#x} CX 


for some 6 > 0. 

The boundary of X is the points in X, which are not interior. 

An interior solution for a linear program is an interior point of the feasible region. 

An interior point method for solving bounded linear programs starts with given interior solutions and 
produces a sequence of interior solutions which converges to an optimal solution. 

An exterior point method for linear programs starts with a point outside the feasible region and 
produces a sequence of points which converges to feasible solution (in the case when the LP is feasible). 


Facts: 


For background reading on the material in this section see [Vas03]. 


1. In linear programming, the problem of finding a feasible solution (i.e., Phase 1) and the problem of 
finding an optimal solution starting from a feasible solution (i.e., Phase 2) can be reduced to each 
other. So, the difference between interior point methods and exterior point methods is not so sharp. 

2. The simplex method, Phase 1, can be considered as an exterior point method which (theoretically) 
converges in finitely many steps. The ellipsoid method [Ha79] is truly an exterior point method. 
Ha¢ijan proved an exponential convergence for the method. 
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3. In the simplex method, Phase 2, we travel from a vertex to an adjacent vertex and reach an optimal 
vertex (if the LP is bounded) in finitely many steps. The vertices are not interior solutions unless 
the feasible region consists of single point. If an optimal solution for a linear program is interior, 
then all feasible solutions are optimal. 

4. The first interior point method was given by Brown in the context of matrix games. The convergence 
was proved by J. Robinson. J. von Neumann suggested a similar method with better convergence. 

5. Karmarkar suggested an interior point method with exponential convergence. After him, many 
similar methods were suggested. They can be interpreted as follows. We modify our objective func- 
tion by adding a penalty for approaching the boundary. Then we use Newton’s method. The recent 
progress is related to understanding of properties of convex functions which make minimization by 
Newton’s method efficient. Recent interior methods beat simplex methods for very large problems. 

6. [Vas03, Sec. A8] On the other hand, it is known that for LPs with small numbers of variables (or, 
by duality, with a small number of constraints), there are faster methods than the simplex method. 

For example, consider an (SRT) with only two variables on top and m variables at the right 
margin (basis variables). The feasible region S has at most m + 2 vertices. Phase 2, starting with 
any vertex, terminates in at most m + 1 pivot steps. 

At each pivot step, it takes at most two comparisons to check whether the tableau is optimal or 
to find a pivot column. Then in m sign checks, at most m divisions, and at most m— 1 comparisons 
we find a pivot entry or a bad column. 

Next, we pivot to compute the new 3m + 3 entries of the tableau. In one division, we find the 
new entry in the pivot row that is not the last entry (the last entry was computed before) and in 
2m multiplications and 2m additions, we find the new entries outside the pivot row and column. 
Finally, we find the new entries in the pivot column in m + 1 divisions. So a pivot step, including 
finding a pivot entry and pivoting, can be done in 8m + 3 operations — arithmetic operations and 
comparisons. 

Thus, Phase 2 can be done in (m + 1)(8m + 3) operations. While small savings in this number 
are possible (e.g., at the (m+ 1)-th pivot step we need to compute only the last column of the 
tableau), it is unlikely that any substantial reduction of this number for any modification of the 
simplex method can be achieved (in the worst case). Concerning the number of pivot steps, for any 
m > 1 itis clearly possible for S to be a bounded convex (m + 2)-gon, in which case for any vertex 
there is a linear objective function such that the simplex method requires exactly |1 + n/2] pivot 
steps with only one choice of pivot entry at each step. It is also possible to construct an (m-+ 2)-gon, 
an objective point, and an initial vertex such that m pivot steps, with unique choice of pivot entry 
at each step, are required. There is an example with two choices at the first step such that the first 
choice leads to the optimal solution while the second choice leads to m additional pivot steps with 
unique choice. 

Using a fast median search instead of the simplex method, it is possible to have Phase 2 done in 
< 100m + 100 operations. 
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51.1 Introduction 


Semidefinite programming, SDP, refers to optimization problems where variables X in the objective 
function and/or constraints can be symmetric matrices restricted to the cone of positive semidefinite 
matrices. (We restrict ourselves to real symmetric matrices, S,,, since the vast majority of applications are 
for the real case. The complex case requires using the complex inner-product space.) An example of a 
simple linear SDP is 


P= min tr CX 
(SDP) subjectto TX=b 
X > 0, 


where T : S, — R”. The details are given in the definitions in section 51.2; the SDP relaxation of the 
Max-Cut problem is given in Example 1 in this section. The linear SDP is a generalization of the standard 
linear programming problem (see Chapter 50), (LP ) min cx, s.t. Ax = b,x > 0, where A € R™*", and the 
element-wise nonnegativity constraint x > 0 is replaced by the positive semidefinite constraint, X > 0. 
These cone constraints are the hard constraints for these linear problems, i.e., they introduce a combinatorial 
element into the problem. In the LP case, this refers to finding the active constraints, i.e., finding the active 
set at the optimum {j : x; = 0}. For SDP, this translates to finding the set of zero eigenvalues of the 
optimum X*. However, for SDP this is further complicated by the unknown eigenvectors. 

But there are surprisingly many parallels between semidefinite and linear programming as well as inter- 
esting differences. The parallels include: elegant and strong duality theory, many important applications, 
and the successful interior-point approaches for handling the hard constraints. The differences include 
possible existence of duality gaps, as well as possible failure of strict complementarity. Since LP is such 
a well-known area, we emphasize the similarities and differences between LP and SDP as we progress 
through this chapter. 
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The study of SDP, or linear matrix inequalities (LMI), dates back to the work of Lyapunov on the stability 
of trajectories from differential equations in 1890, [Lya47]. More recently, applications in control theory 
appear in the work of [Yak62]. More details are given in [BEF94]. 

Cone Programming also called generalized linear programming, is a generalization of SDP [BF63]. Other 
books dealing with problems over cones that date back to the 1960s are [Lue69], [Hol75], and [Jah86]. More 
recently, SDP falls into the class of symmetric or homogeneous cone programming [Fay97], [SA01], which 
includes optimization over combinations of: the nonnegative orthant; the cone of positive semidefinite 
matrices, and the Lorentz (or second-order) cone. 

Positive definite and Euclidean matrix completion problems are feasibility questions in SDP. Research 
dates back to the early 1980s [DG81], [GJM84]. This continues to be an active area of research [Lau98], 
[Joh90]. (More recently, positive definite completion theory is being used to solve large scale instances of 
SDP [BMZ02], [FKM01].) 

Combinatorial optimization applications fueled the popularity of SDP in the 1980s, [Lov79] and the 
strong approximation results in [GW95] and Example 1 in this section. A related survey paper is [Ren99]. 
SDP continues to attract new applications in, e.g., molecular conformation [AKW99], sensor localization 
[Jin05], [SY06], and optimization over polynomials [HL03]. 

The fact that SDP is a convex program that can be solved to any desired accuracy in polynomial time 
follows from the seminal work in [NN94]. 


Examples: 


1. Suppose that we are given the weighted undirected graph G = (V, E) with vertexset V = {1,...,n} 
and nonnegative edge weights w;;, ij € E (with wij; = 0, ij ¢ E). The Max-Cut problem finds 
the partition of the vertices into two parts so as to maximize the sum of the weights on the edges 
that are cut (vertices in different parts). This problem arises in, e.g., physics and VLSI design. We 
can model this problem as a +1 program with quadratic functions. 


= Ly a .x:) = lyT 
es a ne rpare me 4x Lx 
subject to xi =1, Pe Ib seis, 
> Wik ifi= Is 
where L is the Laplacian matrix of the graph, Ljj = 4 k#i We consider the simple 
Wij ifi x ie 
example with vertex set V = {1, 2,3} and nonnegative edge weights wy. = 1,w13 = 2,w23 = 2. 
Pt 
Here, the Laplacian matrix L = |—1 3 —2]. The optimal solution places vertices 1 and 2 
S22. 4 
in one part and vertex 3 in the other; thus it cuts the edges 13 and 23. An optimal solution is 
1 
x= 1] . with optimal value u* = 4. 
—l 


The well known semidefinite relaxation for the Max-Cut problem is obtained using the commu- 
tativity of the trace x’ Lx = tr Lxx’ = tr LX and discarding the (hard) rank one constraint on X. 


wo < v= max ptr LX 
(MCSDP ) subject to diag (X) =e 
X > 0, 
where the linear transformation diag (X) : S, — R” denotes the diagonal of X, and e is the vector 
1 1 -l 
of ones. The optimal solution to MCSDP is X* = | 1 1 —1)]. (This is verified using duality; 
-1 -l ib 


see Example 3 in section 51.4 below.) In fact, here X* is rank one, and column one of X* provides 
the optimal solution x* of MC and the optimal value * = v* = 4. 
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For this simple example, X* was rank one and the SDP relaxation obtained the exact solution of the 
NP-hard MC problem. This is not true in general. However, the MCSDP relaxation is a surprisingly 
strong relaxation for MC. A randomized approximation algorithm based on MCSDP provides a 
+1 vector x with objective value at least .87856 times the optimal value .* [GW94]. Empirical tests 
obtain even stronger results. 


51.2 Specific Notation and Preliminary Results 


Note that in this section all matrices are real. 


Definitions: 


For general m x n matrices M,N € R™”*", we use the matrix inner product (M,N) = tr N!M. The 
corresponding norm is || M||- = tr M™M, the Frobenius norm. 

S,, denotes the space of symmetric n x n matrices. 

The set of positive semidefinite (respectively definite) matrices is denoted by PSD (respectively, PD); 
X > 0 means X € PSD. 


The linear SDP is 
p= min trCX 
(SDP) subjectto TX =b 
X>0, 


where the variable X € S,, the linear transformation T : S, — IR”, the vector b € R”, and the matrix 
C € S,, are (given) problem parameters. The constraint TX = b can be written concretely as tr A,X = bx, 


for some given Ay € S,, k = 1,...,m. Thus, SDP can be expressed explicitly as 
p= min vi Cij Xij 
(SDP) subject to ij AijrXij =, k=1,...,m 
X>0, 


where A;;, denotes the ij element of the symmetric matrix Ax. 

For M € R™*", vec (M) € R™® is the vector formed from the columns of M. 

For S € S,, svec(S) € IR“ =~ is the vector formed from the upper triangular part of S, taken column- 
wise, with the strict upper triangular part multiplied by /2. The inverse of svec is denoted sMat. 

For U € S,, the symmetric Kronecker product is defined by 


s 1 
(M@N)svec (U) = svec (Jesu + MUN") : 
For a linear transformation between two vector spaces T : V > W, the adjoint linear transformation is 


denoted T*4 : W > V, ie,, it is the unique linear transformation that satisfies the adjoint equation for 
the inner products in the respective vector spaces 


(Ty,w)y =v, T%w)y, Vive V, we W. 


If V = W, then this reduces to the definition of the adjoint of a linear operator given in Chapter 5.3. 
We follow the standard notation used in SDP: for y € R", Diag (y) denotes the diagonal matrix formed 
using the vector y, while the adjoint linear transformation is diag (X) = Diag*4(X) € R®. 
For an inner product space V, the polar cone of S C V is 


St = {@: (d,s) > 0, Vs S}. 


Given a set K C R", we let cone (K) denote the convex cone generated by K. (See Chapter 9.1 for more 
information on cones.) A set K is a convex cone if it is closed under nonnegative scalar multiplication and 
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addition, i.e., 
akKCK,Va>0, K+KCK. 
For a convex cone K, the convex cone F C K isa face of K (denoted F < K) if 


xy e K,z=ax+(l—a)yeF,0<a<1 implies xyeFf. 


Facts: 


1. In the space of real, symmetric matrices, S,,, the matrix inner product reduces to (X, Y) = tr XY. 

2. For general compatible matrices M, N, the trace is commutative, ie., tr MN! = trN™M = 
Dien Djn1 Maj. 

3. svec (and sMat) is an isometry between S,, (equipped with the Frobenius norm) and RS 
(equipped with the Euclidean norm). The svec transformation is used in algorithms when solving 
symmetric matrix equations. Using the isometry (rather than the ordinary vec ) provides additional 
robustness. 

4. PSD isa closed convex cone and the interior of PSD consists of the positive definite matrices. The 
following are equivalent:(a) A > 0 (A > 0),(b) thevector ofeigenvaluesA(A) > 0 (A(A) > 0), 
(c) all principal minors > 0 (all leading principal minors > 0). 

5. The Kronecker product K @ L is easily formed using the blocks Kj; L. It is useful in changing matrix 
equations into vector equations (see [HJ94]), (here K is the unknown matrix) vec (NKM!') = 
(M ® N)vec (K). Similarly, the symmetric Kronecker product M@N changes matrix equations 
with symmetric matrix variables U € S,, to vector equations. (See the definition in section 51.2.) 
By extending the definition from S,, to R”*", the symmetric Kronecker product can be expressed 
explicitly using the standard Kronecker product see ([TW05]), 


8M@N = (I+ A)(N@ M' + M@N")+(N@M'+M@N')\(I +A), 


where A is the matrix representation of the transpose transformation. Matrix equations with 
symmetric matrix variables arise when finding the search direction in interior-point methods for 
SDP; see section 51.6. Surprisingly (see [TW05]), for A, B € S,, 


A@B>~0 — > A@B>0; 
A@B> 0 — > A@B>0. 
6. If A € R™*" and the linear transformation Tv = Av, then T*4 = A’. 


7. The polar cone is a closed convex cone and the second polar (K*)*t = K ifand only if K isa closed 
convex cone. 


Examples: 
1, 
1 
2/2 
4 svec (A) =v 


3/2] Alle = lIvil, = V129. 
5/2 
6 


a & N 


3 
5|,v= 
6 
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2. The polars: (a) (R")* = {0}; {0} = R",  (b) for the unit vector e; € R”, {e\}* = {v € R”: 
v; => 0}, (c)(cone{(1 1)7,(1 0)"})t =cone{[O 1]',[1 —1]"}; (dlettT={fveR’: 
v3 < V1V2,V; > 0,Vv2 > 0} be the so-called ice-cream cone [Ber73], i.e., the cone of vectors that 
makes an angle at most 45 degrees with the vector(1 1 1)’.ThenZ =Zt. 


51.3 Geometry 


Extending LP to SDP can be thought of as replacing a polyhedral cone (the nonnegative orthant (RT)") 
by the nonpolyhedral cone PSD. We now see that many of the geometric properties follow through from 
K = (R*)" to K = PSD. 


Definitions: 


A cone K is called self-polar if it equals its polar K = K* 

A face F < K is called facially exposed if F = KN o+, for some d € Kt. 

A face F < K is called projectionally exposed if F is the image of K under an orthogonal projection, 
F = P(K). 


Facts: 

1. Both (R*)" and PSD are closed convex cones with nonempty interior. And both are self-polar 
cones, ie., ((R™)")+ = (R*)", (PSD)*+ = PSD. (See [Ber73].) 

2. Suppose that X € relint F C PSD and rank X = r. Then X = PD, P™, where P is then x r matrix 
with orthonormal columns of eigenvectors corresponding to nonzero (positive) eigenvalues. We 
get that F < PSD if and only if F = PS’, P’. Equivalently, faces F < PSD are characterized by 
{Y e PSD: R(Y) € R(X)}, where X is any matrix in the relative interior of F and R denotes 
the range. Equivalently, we could use N(Y) D N (X), where NV denotes nullspace. (See [Boh48], 
[BC75].) 

3. Both (Rt) and PSD are facially and projectionally exposed, i.e., all faces are both facially and 
projectionally exposed [BW81], [BLP87], [ST90]. 

4. The following relates to closure conditions that are needed for strong duality results. Suppose that 
F < PSD. Then (see [RTW97]), 


PSD + F~ is closed; PSD + Span F is not closed. 


Examples: 

1. Consider the face F = {X € PSD: Xe = 0}, ie., the face of PSD of centered matrices, positive 
semidefinite matrices with row and column sums equal to 0. Let P = (eV) be an orthogonal 
matrix, ie. V'e = 0,V'V = I. Then we get F = V{X € S, 1: X > O}V", ie, a one-to- 
one mapping between the face F and the semidefinite cone in S,_). This relationship is used in 
[AKW99] to map Euclidean distance matrices to positive semidefinite matrices of full rank. 


51.4 Duality and Optimality Conditions 


Duality lies behind efficient algorithms in optimization. Unlike LP, strong duality can fail for SDP if 
a constraint qualification does not hold. We consider duality the linear primal SDP introduced above a 
game-theoretic approach. 


Definitions: 


The corresponding Lagrangian function (or payoff function from primal player X, who plays matrix X, 
to dual player Y, who plays vector y) is L(X,y) = trCX + y"(b — TX). 
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The dual problem is 


d* = maximize b'y 
DSDP subject to Ty <C 
(equivalently Ty + Z=C, Z>O0). 


Weak duality p* > d* holds. If p* = d* and d* is attained, then Strong duality holds. 
Complementary slackness holds if tr ZX = 0; strict complementarity holds if, in addition, Z + X > 0. 


Facts: 


1. The optimal strategy of player X over all possible strategies by the dual player Y has optimal value 
greater or equal to that of the optimal strategy for player Y over all strategies for player X, i.e., 


an Ps : et * T es adj 
p = mina LOC y) st =e y+trxX(C— T*’y). 


The first equality in the min—max problem can be seen from the hidden constraint for the inner 
maximization problem. The inequality follows from interchanging min and max. The hidden 
constraint in the inner minimization of the max—min problem yields our dual problem and weak 
duality. However, strong duality can fail. (See [RTW97].) 

2. For X, Z > 0, complementary slackness tr ZX = 0 is equivalent to ZX = 0. 

3. Characterization of Optimality Theorem 
The primal-dual pair X, (y, Z), with X > 0, Z > 0, is primal-dual optimal for SDP and DSDP if 
and only if 


T4y+Z—-C=0 (dual feasibility) 
(OPT ) TX —b=0 (primal feasibility) 


ZX =0 (complementary slackness). 


4, The theorem in the previous fact is used to derive efficient primal-dual interior-point methods. 

5. Again we note the similarity between the above optimality conditions and those for LP, where 
Z, X are diagonal matrices. However, in contrast to LP, the Goldman—Tucker theorem [GT56] can 
fail, i.e., a strict complementary solution may not exist. This means that there may be no optimal 
solution with X+Z > 0 positive definite; see [Sha00], [WW05]. This can result in loss of superlinear 
convergence for numerical algorithms for SDP. Moreover, in LP, Z, X are diagonal matrices and 
so is ZX. This means that the optimality conditions OPT are a square system. However, for SDP, 
the product of two symmetric matrices ZX is not necessarily symmetric. Therefore, the optimality 
conditions are an overdetermined nonlinear system of equations. This gives rise to many subtle 
algorithmic difficulties. 


Examples: 


1. The lack of closure noted in Fact 4 in section 51.3 is illustrated by the sequence 


del a]- br emess {fo of 


This gives rise to an SDP with a duality gap. Let C = i A and A; = F ‘| . Then a primal 


SDP is 0 = mintrCXs.t. trA)X = 0,X > 0. But the dual program has optimal value —oo = 
max Oy s.t. yA; < C, Le., it is infeasible. 
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0 a 0 0 0 0 0 
2. For fixed a > 0, we use parameters b = ql andC = ]0 0 O|,4,= ]0 1 O|,4= 
0 0 0 0 0 0 
1 0 0 
0 O 1).Then we get the primal and dual SDP pair with a finite duality gap: a = mina@Uj, s.t. 
0 1 0 


y 0 0 a 0 0 
Ux = 0, Uj; + 2U23 = 1, U > 0; 0 = max jy s.t. c V1 rs x c 0 | 


0 » 0 0 0 0 
1 1 -l 

3. We now verify that X* = 1 1 —1} is optimal for MCSDP in Example 1 in Section 51.1. 
14. 1 


The dual of MCSDPis mine’y s.t. Diag (y) > ¢L.Sincey=[1 1 2 |’ is feasible for this dual, 
optimality follows from checking strong duality tr LX* = e'y = 4. Equivalently, one can check 
complementary slackness (Diag (y) — L)X* = 0. 


51.5 Strong Duality without a Constraint Qualification 


Definitions: 
A constraint qualification, CQ, is a condition on the constraints that guarantees that strong duality holds 


at an optimum point. 
Slater’s Constraint Qualification (strict feasibility) is defined as: 4X > 0, TX =b. 


Facts: 
1. Suppose that Slater’s CQ holds. Then strong duality holds for SDP and DSDP, i.e., p* = d* and 


d* is attained. 
2. [BW81] Duality Theorem: Suppose that p* is finite. Let F, denote the minimal face of PSD that 


contains the feasible set of SDP. Consider the extended dual program 


d* = maximize b’y 
DSDP. subject to T%y+Z=C 
ZeEFS. 


Then p* = d* and d* is attained, i.e., strong duality holds between SDP and DSDP.. 
3. An algorithm for finding the minimal face F, defined above is given in [BW81]. 


Examples: 
We can apply the strong duality results to the two examples given in Section 51.4. 
1 
1. For the first example, replace PSD in the primal with the minimal face F = PSDN | | a 
cone{A,};so Ft = {A,}* and the dual is now feasible. 


0 0 077 
2. Forthe second example, replace PSD in the primal with the minimal face F = PSD/N 0 0 0 | ; 
0 0 1 


so Up3 is no longer restricted to be 0 in the dual. 
3. The strong duality results are needed in many SDP applications to combinatorial problems where 
Slater’s constraint qualification (strict feasibility) fails, e.g., [WZ99], [ZKR98]. 
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51.6 A Primal-Dual Interior-Point Algorithm 


As mentioned above, it was the development of efficient polynomial-time algorithms for SDP at the end 
of the 1980s that spurred the increased interest in the field. We look at the specific method derived in 
[HRV96]. (See also [KSH97], [Mon97].) 

We assume that Slater’s constraint qualification (strict feasibility) holds for both primal and dual 
problems. 


Definitions: 


For each x > 0, the log-barrier approach applied to DSDP requires the solution of the log-barrier problem 


DSDP , dj, = maximize bly + plog det Z 
subject to Ta y+ Z=C, Z>0. 


The Lagrangian is L(X,y, Z) = b’y+ log det Z+ tr X(C—T*4 y—Z), where we use X for the Lagrange 
multiplier vector. 


Facts: 


1. The derivative, with respect to (X, y, Z) of the Lagrangian, yields the optimality conditions for the 
barrier problem: 


T4y+Z—-C=0 (dual feasibility) 
OPT TX—b=0 (primal feasibility) 
uwZ~'—X=0 (perturbed complementary slackness). 


We let X,,,Y,Z, denote the unique solution of OPT,,, ie., the unique optimum of the log- 
barrier problem. The set {X,,y,,Z, : & | O} is called the central path. Successful primal-dual 
interior-point methods are actually path-following methods, i.e., they follow the central path to 
the optimum at yz = 0. 

2. The last equation in the optimality conditions OPT,, is very nonlinear and leads to ill-conditioning 
as jt gets close to zero. Therefore, it is replaced by the more familiar 


OPT ,, ZX — ul=0 (perturbed complementary slackness). 
We denote the resulting optimality conditions using 
F,,(X,y, Z) = 0. 


This system has the same appearance as the one used in LP. However, it is an overdetermined 
nonlinear system, ie., F,, : S, x R” x S, > S, x R” x R’*", since the product ZX is not 
necessarily symmetric. Therefore, the successful application of Newton’s method done in LP must 
be modified. A Gauss—Newton method is used in [KMR01]. Another approach is to symmetrize 
the last equation of OPT,, . A general symmetrization scheme is presented in [MZ98]. 

3. We derive and present (arguably) the most popular algorithm for SDP, the HKM method [HRV96], 
[KSH97], [Mon97]. 

We consider a current estimate X,y, Z of the optimum of the log-barrier problem DSDP ,,, 
where both X > 0, Z > 0. Since we want ZX = wl, we set uw = ttr ZX. We use a centering 
parameter 0 < o that is adaptive in that it decreases (respectively increases) according to the 
decrease (respectively increase) in the duality gap estimate at each iteration. Small o signifies an 
aggressive step in decreasing jz to 0. We replace 44 < oy in the optimality conditions. We denote 
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Ra 
the optimality conditions using F,, = F,,(X,y, Z) = Ip | , Le., using three residuals. To derive 
R, 
AX 
the HKM method, we solve for the Newton direction dy = Ay | in the Newton equation 
AZ 
0 Td. T-. Ry 
Fi(%y Z) = [r 0 0 | dy =- A =-F,. 
Z OO -X R, 


To solve the above linearized system efficiently, we use block elimination. We use the first equation 
to solve for 


AZ = —Ry — T*4 Ay. 
We then substitute into the third equation and solve for 
AX =Z" ((Rg+T Ay)X—R,). 


We then substitute this into the second equation and obtain the so-called Schur complement system, 
which is similar to the normal equation in LP: 


TZ1T*4 AyX =—TZ"! (RaX — R.) — fp 
=—-TZ'!(RgX —ZX+pl)—TX+b 
=b—TZ7!(RyX +l). 


This is a positive definite linear system. Once Ay is found, we can backsolve for AX, AZ. 

We now observe another major difference with LP and a major difficulty for SDP. Though A Z is 
now symmetric, this is not true, in general, for AX, since the product of symmetric matrices is not 
necessarily symmetric. Therefore, we symmetrize AX <— }(AX + AX"). We have now obtained 
the search direction (AX, Ay, AZ). 

Next, a line search is performed that maintains X, Z positive definite, i.e., we find the next iterate 
(X, y, Z) using the primal and dual steplengths X <—- X +a,AX > 0,Z <— Z+agAZ > Oand 
set y <— y+ aq Ay. We then update o, jz and repeat the iteration, i.e., we go back to Fact 3 above. 

4. See [HRV96] for more details on the HKM algorithm, including a predictor-corrector approach. 
Details on different symmetrization schemes that lead to various search directions and convergence 
proofs can be found in [MT00]. Algorithms based on bundle methods that handle large-scale SDP 
can be found in [HOO0]. 


51.7 Applications of SDP 


There are a surprising number of important applications for SDP, several of which have already been 
mentioned. Early applications in engineering involved solutions of linear matrix inequalities, LMIs, e.g., 
Lyapunov and Riccatti equations; see [BEF94]. Semidefinite programming plays an important role in 
combinatorial optimization where it is used to solve nonlinear convex relaxations of NP-hard problems 
[Ali95], [WZ99], [ZKR98]. This includes strong theoretical results that characterize the quality of the 
bounds; see [GR00]. Matrix completion problems have been mentioned above. This includes Euclidean 
Distance Matrix completion problems with applications to, e.g., molecular conformation; see [AKW99]. 
Further applications to control theory, finance, nonlinear programming, etc. can be found in [BEF94] and 
[WSV00]. 
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Facts: 
12 
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SDP arises naturally when finding relaxations for hard combinatorial problems. We saw one simple 
case, the Max-Cut problem, MC, in the introduction in Example 1, i.e, MC was modeled as a 
quadratic maximization problem and the binary +1 variables were modeled using the quadratic 
constraints x? = 1. 

In addition to +1 binary variables in Fact 1, we can model 0,1 variables using the quadratic 
constraints xi — x; = 0. Linear constraints Ax — b = 0 can be modeled using the quadratic 
constraint || Ax — b||? = 0. Thus, in general, many hard combinatorial optimization problems are 
equivalent to a quadratically constrained quadratic program, QQP . Denote the quadratic function 
qily) = sy’ Qiyt+y"b; +cj;,y € R",i =0,1,...,m. Then we have 


qg= min oy) 


P 
(QQP) subject to qi(y)=0, i=1,...m. 


(For simplicity, we restrict to equality constraints.) 


. SDParises naturally from the Lagrangian relaxation of QQP in Fact 2 above. We write the Lagrangian 


L(y, x), with Lagrange multiplier vector x. 


quadratic in y linear in y constant in y 
L(y,x) = 577 (Qo-— D> SQ)y + xT o— dy bi) + (ea — SL 4G): 


Weak duality follows from the primal-dual pair relationship 


q* =minmaxL(y,x) > d* = maxmin L(y, x). 
yx x oy 


We now homogenize the Lagrangian by adding yp to the linear term: yoy" (bp — 0", xibi), yp = 1. 
Then, after moving the constraint on yo into the Lagrangian, the dual program becomes 
d* = maxmin L(y,x) 
= maxmin  fy™(Qo— 772, x1 Qi) y(+ty)) 
+ yoy" (bo cae ae x;b;) 
+ (co — So72, xici) (—#). 
We now exploit the hidden constraint from the inner minimization of a homogeneous quadratic 


function, i.e., the Hessian of the Lagrangian must be positive semidefinite. This gives rise to the 
SDP 


(D) s.t. T 


xeR”, teR, 


where the linear transformation T : Rt! —> S,,,, and the matrix B are defined by 


. 0 bg ale -t pee 
~ [bo Qol’ el, ec aebr. Sees Oe) 
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The dual, DD, of the program D gives rise to the SDP relaxation of QQP. 


p’ = inf tr BU 
: —1 
(DD) st. TUS | 
—c 
U>0. 
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52.1 Introduction and Mise-En-Scene 


Linear algebra is used extensively in statistical science, in particular in linear statistical models, see, e.g., 
Graybill [Gra01], Ravishanker and Dey [RD02], Rencher [Ren00], Searle [Sea97], Seber and Lee [SL03], 
Sengupta and Jammalamadaka [SJ03], and Stapleton [Sta95]; as well as in applied economics, see, e.g., 
Searle and Willett [SW01]; econometrics, see, e.g., Davidson and MacKinnon [DM04], Magnus and 
Neudecker [MN99], and Rao and Rao [RR98]; Markov chain theory, see, e.g., Chapter 54 or Kemeny and 
Snell [KS83]; multivariate statistical analysis, see, e.g., Anderson [And03], Kollo and von Rosen [KvR05], 
and Seber [Seb04]; psychometrics, see, e.g., Takane [Tak04] and Takeuchi, Yanai, and Mukherjee [TYM82]; 
and random matrix theory, see, e.g., Bleher and Its [BI01] and Mehta [Meh04]. Moreover, there are several 
books on linear algebra and matrix theory written by (and mainly for) statisticians, see, e.g., Bapat [Bap00], 
Graybill [Gra83], Hadi [Had96], Harville [Har97], [Har01], Healy [Hea00], Rao and Rao [RR98], Schott 
[Sch05], Searle [Sea82], and Seber [Seb06]. 

As Miller and Miller [MM04, p. 1] and Wackerly, Mendenhall, and Scheaffer [WMSO02, p. 2] point 
out: “Statistics is a theory of information, with inference making as its objective” and may be viewed 
as encompassing “the science of basing inferences on observed data and the entire problem of making 
decisions in the face of uncertainty.” 

In this chapter we present an introduction (Section 52.2) to the use of linear algebra in studying 
properties of random vectors (Section 52.3) and in linear statistical models (Section 52.4); for further 
uses, see, e.g., [PSSO5] and [PS05], and for an introduction to the use of linear algebra in multivariate 
statistical analysis see Chapter 53 or [PSS05]. 

We begin (in Section 52.2) with a brief introduction to the basic concepts of statistics and random 
variables [MM04], [WMS02], [WS00], with special emphasis (in Section 52.3) on random vectors — 
vectors where each entry is a random variable. 

All matrices (with at least two rows and two columns) considered in this chapter are nonrandom. 


52-1 


52-2, Handbook of Linear Algebra 


52.2 Introduction to Statistics and Random Variables 


Notation: 


In this chapter most uppercase, light-face italic letters, in particular X, Y, and Z, denote scalar random 
variables, but the notation P(A) is reserved for the probability of the set A. Throughout this chapter, the 
uppercase, light-face roman letter E denotes expectation. 


Definitions: 


The focus in statistics is on making inferences concerning a large body of data called a population based 
ona subset collected from it called a sample. An experiment associated with this sample is repeatable and 
its outcome is not predetermined. A simple event is one associated with an experiment, which cannot 
be decomposed, and a simple event corresponds to one and only one sample point. The sample space 
associated with an experiment is the set of all possible sample points. 

Suppose that S is a sample space associated with an experiment. To every subset A of S we assign a real 
number P(A), called the probability of A, so that the following axioms hold: (1) P(A) > 0, (2) P(S) = 1, 
(3) If Aj, Ay, A3,... forma sequence of pairwise mutually exclusive subsets in S,ie., A; Aj = Wifi F j, 
then P(A; U Aj U A3U---) = 0°, P(Aj). 

A random variable is a real-valued function for which the domain is a sample space. 

A random variable which can assume a finite or countably infinite number of values is discrete. 

The probability P(Y = y) that the random variable Y takes on the value y is defined as the sum of the 
probabilities of all sample points in S that have the value y, and the probability function of Y is the set 
of all the probabilities P(Y = y). 

If the random variable Y has the probability function P(Y = 0) = pand P(Y = 1) = 1— p for some 
real number p in the interval [0, 1], then Y is a Bernoulli random variable. 

The cumulative distribution function (cdf) is F(y) = P(Y < y) of the random variable Y for 
—oo < y < o.A random variable Y is continuous when its cdf F (y) is continuous for —oo < y < 00, 
and then its probability density function (pdf) f(y) = dF(y)/dy. 

Suppose that the continuous random variable Y has pdf f(y) = 1 for0 < y < land f(y) = 0 
otherwise. Then Y follows a uniform distribution on the interval [0, 1]. 

The expectation (or expected value or mean or mean value) E(Y) of the random variable Y is E(Y) = 
ae yP(Y = y) when ¥ is discrete with probability function P(Y = y) and is E(Y) = if oe yf (dy 
when Y is continuous with pdf f(y). 

The variance o” of the random variable Y is 


a” =var(Y) =E((Y — )’), 


where jz = E(Y) and the standard deviation o = Jo”. 


Facts: 

1. The variance o? = var(Y) = E(Y*) — E’(Y). 

2. For any random variable Y, the expectation of its square E(Y) > E?(Y) with equality if and only 
if the random variable Y = E(Y) with probability 1. 

3. IfY isa Bernoullirandom variable with probability function P(Y = 0) = pand P(Y = 1) = 1—p, 
then the expectation E(Y) = 1 — p and the variance var(Y) = p(1— p). 

4. Ifthe random variable Z follows a uniform distribution on the interval [0, 1], then the expectation 
E(Z) = 1/2 and the variance var(Z) = 1/12. 
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Examples: 


1. Every person’s blood type is A, B, AB, or O. In addition, each individual either has the Rhesus (Rh) 
factor (+) or does not (—). A medical technician records a person’s blood type and Rh factor. The 
sample space for this experiment is {A+, B+, AB+, O+, A—, B—, AB—, O—} with eight sample 
points. 

2. Consider the experiment of tossing a single fair coin and define the random variable Y = 0 if the 
outcome is “heads,” and Y = 1 if the outcome is “tails.” Then Y is a Bernoulli random variable, 
and P(Y = 0) = } = P(Y = 1), E(Y) = 3, var(Y) = 3. 

3. Suppose that a bus always arrives at a particular stop in the interval between 12 noon and 1 p.m. 


and that the probability that the bus will arrive in any given subinterval of time is proportional 
only to the length of the subinterval. Let Y denote the length of time that a person arriving at 
the stop at 12 noon must wait for the bus to arrive, and let us code 12 noon as 0 and measure 
the time in hours. Then the random variable Y follows a uniform distribution on the interval 
[0, 1]. 


52.3 Random Vectors: Basic Definitions and Facts 


Linear algebra is extensively used in the study of random vectors, where we consider the simultaneous 
behavior of two or more random variables assembled as a vector. In this section all vectors and matrices 
are real. 


Notation: 


In this section uppercase, light-face italic letters, such as X, Y, and Z, denote scalar random variables and 
lowercase bold roman letters, such as x, y, and z, denote random vectors. Uppercase, light-face italic letters 
such as A and B denote nonrandom matrices. 


Definitions: 


Let A € R"** and B € R"™‘. Then the partitioned matrix [A| B] is the n x (k + q) matrix formed by 
placing A next to B. 

Akx1random vector yisavectory = [Yi,..., Y«] T ofk random variables Y;,..., Y;. The expectation 
(or expected value or mean vector) of y is the k x 1 vector E(y) = [E(Y)),..., E(Y,)]". Sometimes, for 
clarity, a vector of constants (belonging to R”) is called a nonrandom vector and a matrix of constants a 
nonrandom matrix. (Random matrices are not considered in this chapter.) 

The covariance cov(Y, Z) between the two random variables Y and Z is cov(Y, Z) = E ( (Y-—p)(Z— v)) ; 
where « = E(Y) andv = E(Z). 

The correlation (or correlation coefficient or product-moment correlation) cor(Y, Z) between the 
two random variables Y and Z is cor(Y, Z) = cov(Y, Z)/./var(Y )var(Z). 

The covariance matrix (or variance-covariance matrix or dispersion matrix) of the k x 1 random 
vector y = [Yj,...,Yx]7 is the k x k matrix var(y) = © of variances and covariances of all the entries 
of y: 


var(y) = & = [o;;] = [cov(¥;, Y;)] = [E(Y%i — wi)(Yj — 4 )] 
=E((y— w)(y— 4)"), 


where pp = E(y). The determinant det © is the generalized variance of the random vector y. The variances 
oj; are often denoted as o? and, in this, chapter, we will assume that they are all positive. If 0; = 0, then 
the random variable Y; = E(Y;) with probability 1, and then we interpret Y; as a constant. In statistics it is 
quite common to denote standard deviations as 0;. (The reader should note that in all the other chapters 
of this book except in the two statistics chapters, o; denotes the ith largest singular value.) 
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The cross-covariance matrix cov(y, z) between the k x 1 random vector y = [Y;,..., Y;]7 and the 
q x lrandom vector z = [Z},..., Zi is the k x q matrix of all the covariances cov(Y;, Zj);i = 1,...,k 
and j = 1,...,q: 


cov(y, z) = [cov(Y;, Z;)] = [E(¥; — mi)(Z; — »j)] =E((y- w@-v)"), 


where = [y;] = E(y) and v = [v;] = E(z). The random vectors y and z are uncorrelated whenever 
the cross-covariance matrix cov(y, z) = 0. 

The correlation matrix cor(y) = R, say, ofthe k x 1random vectory = [Y),..., Yr] T isthek xk matrix 
of correlations of all the entries in y: cor(y) = R = [p;j] = [cor(¥j, Yj)] = [so], where 0; = ./ojj = 
standard deviation of Y;; 03,0; > 0. 

Let 1, denote the k x 1 column vector with every entry equal to 1. Then J, = 1,1 E is the k x k all-ones 
matrix (with all k* entries equal to 1) and Cy = I, — 7 Jy is the k x k centering matrix. 

Suppose that the real positive numbers pi, p2,..., px are such that a pi = 1.Thenthek x 1 random 
vector y = [Yj,..., Yx]7 follows a multinomial distribution with parameters n and pj, po,..., px if the 
joint probability function of Y), Y2,..., Y; is given by 


P(Y, = 15 Yo = Yau. --> Yk = Ve) = re ae 


yilya!- ste 


where for each i, y; = 0,1,2,...,n and yy yj =n. Whenk = 2 the distribution is binomial, and when 
k = 3 the distribution is trinomial. 

Let the symmetric matrices A € IR*** and B € R***. Then A > B means A— B is positive semidefinite 
and A > B means A — B is positive definite. The partial ordering induced by > is called the partial 
semidefinite ordering (or Loewner partial ordering or Léwner partial ordering). (See Section 8.5 for 
more information.) 


Let the (k + q) x 1 random vector x have covariance matrix &. Consider the following partitioning: 


x= ; ‘ E(x) = a ; var(x) = X= Re | . 


where y and z have k and q elements, respectively. Then the partial covariance matrix X,,., of the q x 1 
random vector z after adjusting for (or controlling for or removing the effect of or allowing for) the 
k x 1 random vector y is the (uniquely defined) generalized Schur complement 
B/ Shy = loys] = Bie= By Diy Dye 

of Xy, in X; any generalized inverse X,,, satisfying Ly, = Ly, XU), Xyy may be chosen. 

When %y, is positive definite, we use the inverse & ee instead of a generalized inverse 2 ye and refer to 
Df Diy = Bag — By is Lyz as the Schur complement of X,, in X. 

The q x 1 random vector e,.y = z— vy — Xz,D ih (y — 4) is the (uniquely defined) vector of residuals 
of the q x 1 random vector z from its regression on the k x 1 random vector y. 

The ijth entry of the partial correlation matrix of q x 1 random vector z = [Zj,..., Z4\" after 
adjusting for the k x 1 random vector y is the partial correlation coefficient between z; and z; after 
adjusting for y: 


Oij-y 
/Oji-yO jj-y 


which is well defined provided the diagonal entries of the associated partial covariance matrix are all 
positive. 


Pij-y = ;1,j=1,...,9q, 
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Facts: 
Le 


15. 


16. 


17. 


[WMS02, Th. 5.13, p. 265] Let the k x 1 random vector y = [Yj,..., Y¢]7 follow a multinomial 
distribution with parameters n and p;,..., px and let the k x 1 vector p = [pi,..., px]?. Then: 


* The random variable Y; can be represented as the sum of n independently and identically dis- 
tributed Bernoulli random variables with parameter p;; i = 1,...,k. 


* The expectation E(y) = np and the covariance matrix 
var(y) = n(diag(p) — pp’ ) = VY, 


say, where diag(p) is the k x k diagonal matrix formed from the k x 1 nonrandom vector p. 


* The covariance matrix WY; is singular since all its row (and column) totals are 0, andthe rank(W;,) = 
k—-1. 


. When the k x 1 multinomial probability vector p = 1,/k, then the multinomial covariance matrix 


Wy = Ch, where C; is the k x k centering matrix. 
The k x k covariance matrix var(y) = © = E(yy") — pp’, where pp = E(y). 
The k x k correlation matrix cor(y) = [diag(X)]~'/? 5 [diag()]~'/?, where © = var(y). 
The k x q cross-covariance matrix 
cov(y,z) = Ly, = xy = (cov(z,y))' = E(yz’) — pv’, 
where pt = E(y) and v = E(z). 
[RM71, Lemma 2.2.4, p. 21]: The product AB~C (for A 4 0, C ¥ 0) is invariant with respect to 
the choice of B7 <=> range(C) © range(B) and range(A‘) C range(B7). 
Consider the (k + q) x (k + q) covariance matrix 


r= Lyy Lye 
Lzy Xzz 
Then the range (or column space) range(X,,) © range(X,,) and range(X,,) C range(Z,,), and, 


hence, the matrix X,, Big x,, and the generalized Schur complement &/Zyy = Xz, — Lzy Zyy Lyz 


are invariant (unique) with respect to the choice of generalized inverse X,, 


. Let the k x 1 random vector x = [X,...,X%]7. Then the k x 1 centered vector Cyx = [X; — 
X,...,X% — X]", where C; is the k x k centering matrix and the arithmetic mean (or average) 
x k 
X = Dijai Xi/k. 


A nonsingular positive semidefinite matrix is positive definite. 


. A covariance matrix is always symmetric and positive semidefinite. 

. A cross-covariance matrix is usually rectangular. 

. A correlation matrix is always symmetric and positive semidefinite. 

. The diagonal entries of a correlation matrix are all equal to 1 and the off-diagonal entries are all at 


most equal to 1 in absolute value. 


. [PSO5, p. 168] The (generalized) Schur complement of the leading principal submatrix ofa positive 


semidefinite matrix is positive semidefinite. 

Let y bea k x 1 random vector with covariance matrix ©. Then the variance var(a’y) = a’ Za 
for all nonrandom a € R*. (Since variance must be nonnegative this fact shows that a covariance 
matrix must be positive semidefinite.) 

Let y beak x 1 random vector with expectation pp = E(y), covariance matrix 2 = var(y), and 
let the matrix A € R”** and the nonrandom vector b € R". Then the expectation E( Ay + b) = 
AE(y) + b = Ap + band the covariance matrix var( Ay + b) = Avar(y) AT = ADA’, 

Let y beak x 1 random vector with expectation js = E(y), covariance matrix = var(y), and let 
the matrix A € R***, not necessarily symmetric. Then E(y? Ay) = wp Aw + tr(AX). 
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18. [Rao73a, p. 522] Let y beak x 1 random vector with expectation 2 = E(y) and covariance matrix 
X = var(y), and let [pe| &] denote the k x (k + 1) partitioned matrix with pz as its first column. 
Then y — p € range(X) andy € range([jz| Z]), both with probability 1. 

19. Let the (k+q) x 1 random vector x have covariance matrix ©. Consider the following partitioning: 


y= Lyy Lyz 
igs. Dae |* 


where y and z have k and q components, respectively. Then 


* The variance var(a’y + b’z) = a? Dyya + 2a’ Dy-b + b’z_,b for all nonrandom a € R* and 
for all nonrandom b € R’. 


* The covariance matrix var(Ay + Bz) = AX,,A™ + AL,,B’ + BX,,A! + BY,,B! for all 
Ae R™* andall B € R"™4, 


* [PS05b, pp. 187-188] For any A € R1** the covariance matrix 
var(z — Ay) > var(z — Ley Ly) 
with respect to the partial semidefinite ordering, and the partial covariance matrix 
vale Dy 9) = lps Byd,, Dye = Tey = Dl Lys 


the generalized Schur complement of X,, in X. 


* Let q = k. Then the covariance matrix var(y + z) = var(y) + var(z) if and only if cov(y,z) = 
—cov(z,y), Le., the cross-covariance matrix cov(y,z) is skew-symmetric; the condition that 
cov(y, Z) = 0 is sufficient, but not necessary (unless k = 1). 


* The vector &,,%),y is not necessarily invariant with respect to the choice of generalized inverse 
Dy» but its covariance matrix var(Zzy yy) =X,% i &,, is invariant (and, hence, unique). 
Examples: 


1. Let the 4 x 1 random vector x have covariance matrix &. Consider the following partitioning: 


1 0aé ib 
0 1 d 
_ He oF 
Z ac 1 0 
b dol 


where y and z each have 2 components. Then var(y+ z) = var(y) + var(z) ifand only ifa = d =0 
andc = —b, with b? < 1. 

2. Let the k x 1 random vector y = [Y,,..., Y,]7 follow a multinomial distribution with parameters 
nandp = [p1,..., pel’, with p; +---+ px = land p; > 0,..., pe > 0, and let the k x k matrix 
A = [a;;]. Then the expectation E(y! Ay) =n(n—1)p’Ap+n an Aji Pi - 

3. Let the 3 x 1 random vector y = [Yj, Yo, Y3]" follow a trinomial distribution with parameters n 
and pj, pa, p3, with p; + po + p3 = land p; > 0, po > 0, p3 > 0, and let the 3 x 1 vector 
Pp = [Pi 2 p3]?. Then: 


* The expectation E(y) = n[p1, po, ps]? and the covariance matrix 


pill — pi) —Ppip2 — pips 
W3 = var(y) =n Pip2 p2(1 — p2) —prp3 |> 
— Pips —prps —-p3(1— ps) 
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which has rank equal to 2 since W3 is singular and the determinant of the top left-hand corner of 
W; equals np; po p3 > 0. 


* The partial covariance matrix of Y, and Y) adjusting for Y3 is the Schur complement W3/ ( p3ad- 
Ps) = nS, say, where 


eae pid— pi) —pip2 
—Pip2 p2(1 — p2) 


_ | Pips 1 Leap pep Pipe E | 
—pops| Ps — ps) pepe [-1 1? 


which has rank equal to 1, and so rank(W3) = 2. 


¢ When p; = po = p3 = 1/3, then the covariance matrix Y3 = (n/3)C3 and the partial covariance 
matrix of Y, and Y> adjusting for Y3 is (n/3)C»; here, C;, is the h x h centering matrix, h = 2,3. 


4. [PSO5, p. 183] Ifthe 3 x 3 symmetric matrix 


l sr r 
R 113 ee 
R3= |r 1 3) =], 


ry 1 


is a correlation matrix, then rj < 1 foralll <i <j < 3. But not all symmetric matrices with 
diagonal elements all equal to 1 and all off-diagonal elements r;; such that r?, < 1 are correlation 
matrices. For example, consider R3 with r?, < 1 and r3, < 1. Then R; is a correlation matrix if 


and only if 
rts — 4/ (1 — ri3)(1 — 733) < ri < rista3 + 1/1 — 3) — 133). 


When r}3 = 0 and rj2 = 123 = 1, say, then this condition becomes r? < 1/2 and so the matrix 


is not a correlation matrix. 
When r?, < 1, then the matrix R; is a correlation matrix if and only if any one of the following 
conditions holds: 
* det(R3) =1—r?, —1r?, — 13, + 2rprisra3 = 0. 
* (i) m € range(R) and (ii) 1 > rt R; rt for some and, hence, for every generalized inverse 
R5. 


5. Let the random vector x be 2 x 1 and write 


2 
x= y ‘. E(x) = a ; var(x) = Y= p A ; 


with a; > 0. Then the residual vector e,.y = z—v — Xz,D ie (y — p) of the random vector z from 
its regression on y becomes the scalar residual 


oO v4 
ezy =Z—Vv 7 (Y — p) 
oy 
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of the random variable Z from its regression on Y. The matrix of partial covariances of the random 
vector z after adjusting for y becomes the single partial variance 


2 2 ye 2 2 
ony = 0, 2 =o;(1 Pyz) 
Fy 


of the random variable Z after adjusting for the random variable Y; here, the correlation coefficient 


Pyz = Oyz/(FyGz). 


52.4 Linear Statistical Models: Basic Definitions and Facts 


Notation: 


In this section, the uppercase, light-face italic letter X is reserved for the nonrandom n x p model matrix 
and V is reserved for an n x n covariance matrix. The uppercase, light-face italic letter H is reserved for the 
(symmetric idempotent) n x n hat matrix X(X T X)~X? and M = I — H is reserved for the (symmetric 
idempotent) n x n residual matrix. The lowercase, bold-face roman letter y is reserved for an observable 
n x 1 random vector and x is reserved for a column of the n x p model matrix X. 


Definitions: 


The general linear model (or Gauss—Markov model or Gau§B—Markov model) is the model 
M = ty, XB, o’V} 


defined by the equation y = X3 + e, where E(y) = XG, E(e) = 0, var(y) = var(e) = o’V. The vector 
y is an n x 1 observable random vector, € is an n x 1 unobservable random error vector, X is a known 
nx p model matrix (or design matrix, particularly when its entries are —1,0, or +1), Bisa p x 1 vector 
of unknown parameters, V is a known n x n positive semidefinite matrix, and o? is an unknown positive 
constant. The realization of the n x 1 observable random vector y will also be denoted by y. 

The classical theory of linear statistical models covers the full-rank model, where X has full column 
rank and V is positive definite. In the full-rank model, the ordinary least squares estimator 


OLSE(@) = 3 = (X7X)1X'y= Xly 
and the generalized least squares estimator (or Aitken estimator) 
GLSE(B) = @ = (X'V7!X) 1} XT Vly, 


where X? denotes the Moore—Penrose inverse of X. 

When either X or V is (or both X and V are) rank deficient, then it is usually assumed that rank(X) < 
rank(V). The model. 4 = {y, XB, o*V} is called a weakly singular model (or Zyskind—Martin model) 
whenever range(X) C range(V), and then rank(X) < rank(V), and is consistent if the realization y 
satisfies y € range([X | V]). 

Let 3 be any vector minimizing ||y — XG|? = (y — XB)" (y — XB). Then ¥ = XG = OLSE(XB) = 
the ordinary least squares estimator (OLSE) of X. When rank(X) < p, then B is an ordinary least 
squares solution to ming(y — X3)"(y — X). Moreover, B is any solution to the normal equations 
X™XG = X'y. The vector of OLS residuals is e = y — = y — X( and the residual sum of squares 
SSE =e'e=(y—y)'(y-Y). 

The coefficient of determination (or coefficient of multiple determination or squared multiple 
correlation) R? = 1 — (SSE/yC,y) identifies the proportion of variance explained in a multiple linear 
regression where the model matrix X = [1,, | Xt1) | --- |X{p—1)] with p — 1 regressor vectors (or regres- 
SOrs) X{1],...,X{p—1] each n Xx 1. In simple linear regression p = 2 and the model matrix X = [1,,| x] 
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with the single regressor vector x. The sample correlation coefficient r = x’ Cry/ x? C,x-y? Crys 
where it is usually assumed that x is an n x 1 nonrandom vector (such as a regressor vector) and y is a 
realization of the n x 1 random vector y. 

Let the matrix A € R**" and let the matrix K € R**?. Then the linear estimator Ay is a linear 
unbiased estimator (LUE) of K@ if E(Ay) = K@ for all 8 € R?. Let the matrix B € R**". Then the 
LUE By of K is the best linear unbiased estimator (BLUE) of K @ if it has the smallest covariance 
matrix (in the positive semidefinite ordering) in that var( Ay) > var( By) for all LUEs Ay of KG. 

The hat matrix H = X(X7X)~ X7 associated with the model matrix X is so named since y = Hy. 
The residual matrix M = I — H and vector of OLS residuals ise = y— Y = y — Hy = My. Let the 
nonrandom vector a € R”. Then the linear estimator a’y, which is unbiased for 0, i.e., E(a’y) = 0,isa 
linear zero function. 

The Watson efficiency ¢ under the full-rank model 4 = {y, Xf, a’ V}, with the n x p model matrix 
X having full column rank equal to p < n and with the n x n covariance matrix V positive definite, 
measures the relative efficiency of the OLSE(3) = vs. the BLUE(3) = G and is defined by the ratio of 
the corresponding generalized variances: 


_ det[var(3)] _ det?(X7X) 
~ det[var(@)]  det(X7VX) - det(X™V-!X) * 


The Bloomfield—Watson efficiency y under the general linear model W = {y, XB,0°*V} with no 
rank assumptions measures the relative efficiency of the OLSE(X) = X{ vs. the BLUE(3) = G and is 
defined by: ¥ = $||HV — VH|? = ||HVM|/, where the norm || Al] = tr'/?(A7 A) is defined for any 
k x q matrix A. 

Then x n covariance matrix (1—p)In+p1,17 = (1—p)I,+ J, has intraclass correlation structure 
(or equicorrelation structure) and is the intraclass correlation matrix (or the equicorrelation matrix). 
The parameter p is the intraclass correlation (or intraclass correlation coefficient). 


Facts: 


The following facts, except for those with a specific reference, can be found in [Gro04], [PS89], or [SJ03, 
§4.1-4.3]. Throughout this set of facts, X denotes the nm x p nonrandom model matrix. 


1. The hat matrix H = X(X'X)~X? associated with the model matrix X is invariant (unique) 
with respect to choice of generalized inverse (X’X)~ and is a symmetric idempotent matrix: 
H = H! = H?’, and rank(H) = tr(H) = rank(X). Moreover, the hat matrix H is the orthogonal 
projector onto range(X). 

2. Ifthe p x p matrix Q is nonsingular, then the hat matrix associated with the model matrix XQ 
equals the hat matrix associated with the model matrix X. 

3. The residual sum of squares SSE = y’ My = (y — ¥)"(y— 9) = y'y — y’ XB, where M is the 
residual matrix and 3 = OLSE(). 


4. In simple linear regression the coefficient of determination R? = r’, the square of the sam- 
ple correlation coefficient. In multiple linear regression with model matrix X = [1,| Xo] = 
(1,1 Xi) | +++ | X{p—1)] and (p — 1) x 1 nonrandom vector a € R?, 

(a? X?C,y) 


R? = maxr2 = max 5 
a" a al XIC,Xoa-y™ Cry 


the square of the sample correlation coefficient r, between the variables whose observed values are 
in vectors y and Xoa. 

5. The vector X3 is invariant (unique) with respect to the choice of least squares solution 3, but 
is unique if and only if X has full column rank equal to p < n, and then @ = OLSE(@) = 
(XTX) xTy = Xty, where X? is the Moore—Penrose inverse of X. The covariance matrix 
var(3) = 02(X?X)-!XTVX(XTX), 


52-10 Handbook of Linear Algebra 


6. The Watson efficiency @ is always positive, and @ < 1 with equality if and only if OLSE(G) = 
BLUE({). 
7. [DLLO2, p. 477], [Gus97, p. 67] Bloomfield-Watson—Knott Inequality. The Watson efficiency 


o= 


det*(X7 X) Pr AA An itt 

det(X7V xX) - det(X7V-!X) =H; Re Iya 
for all n x p model matrices X with full column rank p. Herem = min(p,n—p)anda, >--- > Any 
denote the necessarily positive eigenvalues of the n x n positive definite covariance matrix V. The 
ratios 4A;Ay—j41/(Ai + An—i+1)” in the lower bound for the Watson efficiency are the squared 
antieigenvalues of the covariance matrix V. 

8. [DLL02, p. 454] Let p = 1 and set the n x 1 model matrix X = x. Then the Bloomfield—Watson— 
Knott Inequality is the Kantorovich Inequality (or Frucht—Kantorovich Inequality): 


(x! x)? 2 AhiAn 
x? Vx-xTV-lx — (A, $A,)2’ 


where A; and i, are, respectively, the largest and smallest eigenvalues of the n x n positive definite 
covariance matrix V. 
9. The Bloomfield—Watson efficiency 


1 
v= slHV VA|" = ||HVM|? = tr(HVMVB#) = tr(HVMV) 
=tr(HV* — HVHV) = tr(HV’) — tr((HV)’) > 0 


with equality if and only if OLSE(@) = BLUE(@) if and only if the Watson efficiency ¢ = 1. 

10. [DLL02, p.473] The Bloomfield—Watson Trace Inequality. Let Abe a nonrandom symmetric n x n 
matrix, not necessarily positive semidefinite. Then for all the nonrandom matrices U € R"*? that 
satisfy U'U = Ip: 


min(p,n—p) 
1 
tr(U? A2U) — tr((UT AU)’) <- S- (aj — On—it1)s 


i=1 


nN 


where a > --- > @, denote the eigenvalues of the n x n matrix A. 
11. The Bloomfield—Watson efficiency 


min(p,n—p) 


S- (Aj — An—ita)*s 


i=1 


y =tr(HV’) — tr((HV)’) < 


a 


for all n x n hat matrices H with rank p (and so for all n x p model matrices X with full column 
rank p). Here, A; > --- > A, denote the necessarily positive eigenvalues of the n x n positive 
definite covariance matrix V. 

12. The n x n intraclass correlation matrix Ri. = (1 — p)In — pl,lt has eigenvalues 1 — p with 
multiplicity n — 1 and 1 + p(n — 1) with multiplicity 1, and so Rj. is singular if and only if 
p=-—l/(n—l)orp=1. 

13. The intraclass correlation coefficient p is such that —1/(n — 1) < p < 1 and then x n intraclass 
correlation matrix is positive definite if and only if—1/(n-—1)<p <1. 

14. The inverse of the n x n positive definite intraclass correlation matrix 


7 1 p 
l=ph=p1A!) = E 1,1! ). 
(1 — p)In ~ plnty) + ( 1+p(n—-1)"" 
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15. 


16. 


17. 


18. 


19. 


20. 


21. 


22. 


23. 


Gauss—Markov Theorem (or Gaufs—Markov Theorem). In the full-rank model {y, Xf, o*V}, 
the generalized least squares estimator 3 = GLSE(@) = (X'V~!X)-!X'V~ly = BLUE({). 
In the full-rank model {y, XG, o7I}, the ordinary least-squares estimator OLSE(G) = B = 
(X7X)-1XTy = Xty = BLUE((). 

In the model {y, X, o*V}, where V is positive definite, but with X possibly with less than full 
column rank, the 


BLUE(X) = X(X'V7!X)-X'V7y. 


[Sea97, §5.4] Let the matrix K € R**?. Then K{@ is estimable <> J matrix A € R™*: K? = 
XTA & > range(K‘) Cc range(X/) —_=> KB is invariant for any choice of B = (X7X)~ X7y. 
[Rao73b, p. 282] Consider the general linear model {y, XG, o*V}, where X and V need not be 
of full rank. Let the matrix G € R”*". Then Gy = BLUE(XB) <== G[X|VM] = [X|0], 
where the residual matrix M = I — H. Let the matrix A € R**" and the matrix K € R‘*?. Then 
the corresponding condition for Ay to be the BLUE of an estimable parametric function K G is 
A[X| VM] = [K |0]. 

Let G, and G, both ben x n. If Gy and Gy are two BLUEs of XG under the model {y, XG, o*V}, 
then G;y = Goy for ally € range([X | V]). The matrix G yielding the BLUE is unique if and only 
if range([X | V]) = R". 

Every linear zero function can be written as b? My for some nonrandom b € R". Let the matrix 
G € R"*". Then an unbiased estimator Gy = BLUE(X{) if and only if Gy is uncorrelated with 
every linear zero function. 

[Rao71] Let the matrix A € R"*". Then the linear estimator Ay = BLUE(XQ) under the model 
{y, XB, o*V} if and only if there exists a matrix A so that A is a solution to Pandora’s box 


els) -[e 


[Rao71] Let the (n+ p) x (n+ p) matrix B be defined as any generalized inverse: 


VoX)>. | Be Be 
~ Txt of |B; —Byl” 


Let k’ G be estimable; then the BLUE(k’ 3) = k’@ =k’ By = k’ Bsy, the variance var(k’ 3) = 
o?k? Byk, and the quadratic form y’ B,y/f isan unbiased estimator ofa? with f = rank([V | X])— 
rank(X). 

[PS89] In the model {y, XG, o’V} with no rank assumptions, the OLSE(X) = BLUE(X{) if and 
only if any one of the following equivalent conditions holds: 


°° AV=VH. 
°° HV=HVH. 
* HVM=0. 


* X™VL =0, where then x | matrix L has range(L) = range(M). 

* range(VX) C range(X). 

* range(VX) = range(X) M range(V). 

* HVH < V,ie., V — HVH is positive semidefinite. 

¢ rank(V — HVH) = rank(V) — rank(HVH). 

¢ rank(V — HVH) = rank(V) — rank(VX). 

* range(X) has a basis consisting of r eigenvectors of V, where r = rank(X). 


* V can be expressed as V = aI + XAX? + LBL’, wherea € R, range(L) = range(M), and 
the p x p matrices A and B are symmetric, and such that V is positive semidefinite. 
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More conditions can be obtained by replacing V with its Moore—Penrose inverse V' and the hat 
matrix H with the residual matrix M = I — H. 

24. Suppose that the positive definite covariance matrix V has h distinct eigenvalues: Ay1) > Ap} > 
+++ > Any > Owith multiplicities m,..., mm), aan m; = n, and with associated orthonormalized 
sets of eigenvectors Uj;,..., Ujny, respectively, n x m),...,n x m),. Then OLSE(X) = BLUE(XB) 
if and only if any one of the following equivalent conditions holds: 


° rank(U;}) X) feet rank(U/j,, X) = rank(X). 
* Uj, HU) = (Us, HUG)’ for alli = 1,...,h. 
* Ug AU yy = Oforallt A 93 6.9 = 1.25 h. 


25. [Rao73b] Let the p x p matrix U be such that the n x n matrix W = V+ XUXT™ has range(W) = 
range([X | V]). Then the BLUE(X) = X(X’W~X)-X?Wry. 

26. When V is nonsingular, the n x n matrix G such that Gy is the BLUE of XG is unique, but 
when V is singular this may not be so. However, the numerical value of BLUE(X) is unique with 
probability 1. 

27. [SJ03, §7.4] The residual vector associated with the BLUE(X{9) is 


@=y— X@ = VM(MVM) My = My+ HVM(MVM) My, 


which is invariant (unique) with respect to choice of generalized inverse (MV M)~. The weighted 
sum of squares of BLUE residuals, which is needed when estimating o”, can be written as 
(y — XB)" V-(y— XB) =@'V-é = yy’ M(MVM)- My. 


Examples: 1 1 
1. Letn = 3 and p = 2 with the model matrix X = | 1 0| . Then X has full column rank equal 
1 1 


to 2, the matrix X7 X is nonsingular, and the hat matrix is 


—l 
1 
H = X(X'X)-X? = X(X? x) 1x? = zc] 22 2 
-—1 5 
with rank(H) = tr(H) = 2. The OLSE() is 
1 
‘ 31 + yo + 3) 
B= (XTX) | xly= 3 
5("1 — ¥3) 
where y = [y1, 2, ¥3] 7. The vector of OLS residuals is 
qs SET Tg 1 
1 1 
sal —2 4 —2] Jy] = ACA 2y2 + y3) |-2 
1 <2 “Ll [y, 1 


with residual sum of squares SSE = (y, — 2y2 + y3)?/6. 
Now let the variance 0” = 1 and let the covariance matrix 


Random Vectors and Linear Statistical Models 52-13 
with 5 > 0. Then V is positive definite and 


BLUE() = GLSE(8) = 8 = apg (51 + y2 + dys) 
3("1 — ys) 


while the covariance matrices are 


NIF © 


Hence, the Watson efficiency 


96 


oak 


with equality if and only if 5 = 1. As 6 — 0 or 6 — oo, the Watson efficiency @ — 0. 

Since the eigenvalues of V here are 1 (multiplicity 2) and 6 (multiplicity 1), we find that the 
lower bound for ¢ in the Bloomfield—Watson—Knott Inequality [here, m = min(p,n — p) = 
min(2, 1) = 1] is equal to 45/(1+5)’, and it is easy to show that this is less than 96 / ((28 +1)(6+2) 
(unless 6 = 1, and then both these ratios are equal to 1). 

The Bloomfield—Watson efficiency 


y= 50-8 z0 


with equality if and only if 6 = 1. As 5 — 0 the Bloomfield—Watson efficiency y — 2/9, and as 
5 — oo the Bloomfield—Watson efficiency yy — oo. We note that when 6 = 0, then » is singular, 
but y is well-defined and equal to 2/9. We find that the upper bound for y in the Bloomfield— 
Watson Trace Inequality is (1 — 5)?/4 and certainly (1 — 5)?/4 > 2(1 — 5)*/9 with equality if and 


only if 5 = 1. 
1 
2. Let n = 3 and p = 1 with the model matrix X = x = | 2] and with G = 8, a scalar. The hat 
3 
matrix 
[! 2 3) 
1 1 
He XXX) XK Sak 2 46 
x’x 14 
E 6 | 
and so 


iT 
7 xy yi + 2y2 + 3y3 
B = OLSE(B) = p= ; 
x'x 14 

where the realization y = [y1, y2, y3]". 

Let the covariance matrix have intraclass correlation structure: 


1 p p 
V=!p 1 ¢ 
pp i 


with —1/2 < p < 1. Then V is positive definite and its inverse 


; [i+ —p =P 
p|. 


~ = p)+ 20) | iy Aare | 
—/p —p I1+p 


=] 
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And so 
E x’V-ly = (1 —4p)y1 + 2(1 — p) yo + 393 
= BLUE = GLSE = = | 
B (B) (B) <Tyox 6 —p) 
The variances are (with o? = 1): 
a 1 (1 — p)(14+ 2p) : x'Vx 7+4+1lo 
var(B) = Zs 2a var(B) = Gay oR 


and so the Watson efficiency 


var(B) — 49(1 — p)(1+ 2p) 
~ var(B) (7 —4p)(7 + 11) 


as p — —1/2 oras p —> 1. As p > 0 the Watson efficiency @ > 1. 

Since the eigenvalues of V here are 1 — pe (multiplicity 2) and 1 + 2¢ (multiplicity 1), we 
find that the lower bound for @ in the Bloomfield—Watson—Knott Inequality (which is now the 
Kantorovich Inequality) is 4(1 — p)(1 + 2p)/(2 + py’ and it is easy to show that this is less than 
¢=491-—p)\1+ 2p)/((7 —Ap)(7+ 11p)) (unless p = 0 and then both these ratios are equal 
to 1). 

The Bloomfield—Watson efficiency = 547/49, which is well defined for all p; when p = 
—1/2, then y = 27/98 and when p = 1, then wy = 54/49. We find that the upper bound for w 
in the Bloomfield—Watson Trace Inequality is 97/4 and certainly 907/4 > y = 5497/49, with 
equality if and only if p = 0. 

3. [DLLO2, p. 475] Let A be a nonrandom symmetric n x n matrix, not necessarily positive semidef- 
inite, and let the nonrandom vector u = [u,...,u,]7 be such that uu = 1. Then from the 
Bloomfield—Watson Trace Inequality with U = u, we obtain the special case 


2 ik 
uw Au (u’ Au) < qu = On)’, 
where a and aq, are, respectively, the largest and smallest eigenvalues of A. Now let a,..., ay 


denote n nonrandom scalars, not necessarily all positive, and let a4 = pean a;/n. Then the 
Popoviciu—Nair Inequality 


1 1 
Sai a) < (Amax Asiin) 
- 4 
1=1 
follows directly from the special case above of the Bloomfield—Watson Trace Inequality with u = 
1/,/n and A = diag{a;}. 
4, When the covariance matrix V has intraclass correlation structure, then the OLSE(X 3) = BLUE 
(XB) if and only if X'1,, = 0 or Xf = 1, for some nonrandom p x 1 vector f, and so OLSE 
(XB) = BLUE(X£) when the columns of X are centered or when 1,, € range(X) as in Example 1 


1 1 
above, where X = | 1 0| with first column equal to 13. 
Lt xt 
5. Let the nm x p model matrix X = [1| Xo], where 1 is the n x 1 column vector with every entry 


equal to 1 and Xo isn x (p — 1). Then the hat matrix associated with X, 
1 
A= 1,1 + Cy Xol Xe CX) Xe Ca 
n 


coincides with the hat matrix associated with X, = [1,|C,Xo], since X, = XQ, with the p x p 


Sr 
matrix Q = E q | for some q. 
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When p = 2, we may write Xp) = x, ann x 1 vector, and if X now has rank equal to 2, then 
x’ C,x > 0 and the hat matrix 


H = +1,1% + —c,m"C Int Cx" C 
7 nity TC,x n n n TC,x n n 
And so the quadratic forms 
T 2 T 2 
T _2 ( Cry) T T (x C,y) 
Hy= | 5 My=y Ciy4 , 
y ny=ny TC,x Youve: Cs 


where 7 = >), yi/n = ljy/n. 
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Vectors and matrices arise naturally in the analysis of statistical data and we have seen this in Chapter 52. 
For example, suppose we take a random sample of n females and measure their heights x; (i = 1,2,...,1), 
giving us the vector x = [x),X2,...,Xy] T We can treat x as simply a random vector, which can give rise 
to different values depending on the sample chosen, or as a vector of observed values (the data) taken 
on by the random vector from which we calculate a statistic like a sample mean (average). We use the 
former approach when we want to make inferences about the female population. In this case, we find 
that the value of a given x; will depend on how it varies across the population, and this is described by 
its statistical “distribution,” which is defined by a univariate function of one variable called a probability 
density function (pdf). For example, x; may follow the well-known normal distribution, which seems to 
apply to many naturally occurring measurements. This distribution has a probability density function 
totally characterized by two (unknown) parameters, the population mean yz and the population variance 
o*, where o is the standard deviation; we write x; ~ Nj(u,07), where “~” means “distributed as.” 
(Throughout this chapter o will always refer to a standard deviation and not to a singular value.) When 
the sample is random, the choice of any female does not affect the choice of any other, so that technically 
we say that the x; are statistically independent and they all have the same distribution. 

Three important aspects of statistical inference are: (1) estimating jz from the data and deriving the 
distributional properties of the estimate from the assumed underlying pdf; (2) finding a so-called confi- 
dence interval for jz, which is an interval containing the true value of jz with a given probability, e.g., 0.95 
or, when typically expressed as a percentage, 95%; and (3) testing a hypothesis (called the null hypothesis 
Ho) about whether js is equal to some predetermined value jo, say. Symbolically, we write Ho : 4 = [Mo. 
To test Hp, we need to have a test statistic and be able to derive its distribution when Hp is true and when 
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it is false, called the noncentral distribution in the latter case. Noncentral distributions are usually quite 
complex, but they have a role in determining how good a test is in detecting departures from the null 
hypothesis. 

In multivariate analysis we are interested in measuring not just a single characteristic or variable on 
each female, but we may also want to record weight, income, blood pressure, and so on, so that each x; 
gets replaced by a d-dimensional vector x; of d variables. Then x from the sample of recorded heights 
gets replaced by a matrix X with rows xj. In this case the probability density function for x; is said to 
be multivariate and the population mean is now a vector pt, say. We can look at X as either a matrix 
of numbers, or a random matrix, which we use for carrying out statistical inferences. In doing this, the 
univariate normal distribution extends naturally to the multivariate normal distribution, which has a pdf 
totally characterized by its mean ws and its covariance matrix var(x) = &. One can then derive other 
distributions based on various functions of X, which can then be used for inference. 

The vectors x; can be regarded as a cluster of points in R*. Such a set of points can also come from 
sampling a mixed population of men and women so that there would be two overlapping clusters. Alter- 
natively, the population of females could be a mixture of races, each with its own characteristics, leading 
to several overlapping clusters. In order to study such clusters, we need techniques to somehow reduce 
the dimension of the data (hopefully, to two dimensions) in some optimal fashion, but still retain as 
much of the original variation as possible. We will introduce just three dimension-reducing techniques 
below. The first method, called principal component analysis, is used for a single cluster. The second 
method, called discriminant coordinates, is used for several clusters and the reduction in dimension 
is carried out to maximize group separation. The third method, called canonical variates, is also used 
for a single cluster, but utilizes the internal variation within the vectors. Looking at how one variable 
within a vector depends on the other variables lends itself to a range of other techniques, such as mul- 
tivariate linear regression where we might compare the internal relationships in one cluster with those 
in others. For example, comparing how blood pressure depends on other variables for both males and 
females. 

We finally introduce just one other topic called metric multidimensional scaling. Instead of having 
observations on n people or objects, we simply have measures of similarity or dissimilarity between each 
pair of objects. The challenge then is to try and represent these objects as a cluster in a low-dimensional 
space so that the inter-point Euclidean distances will closely reproduce the dissimilarity measures. Once 
we have the cluster we can then examine it to try and uncover any underlying structure or clustering of 
the objects. 

It is hoped that these few applications will demonstrate the richness of the subject and its interplay 
between statistics and matrices. In this chapter we assume some of the basic statistical ideas defined in 
Chapter 52. However, there has to be a change in notation as we now need to continually distinguish 
between random and nonrandom vectors and matrices. 


53.1 Data Matrix 


Notation: 


The latter part of the alphabet from u to z, upper or lower case, together with Q we reserve for random 
quantities, and the remainder of the alphabet for nonrandom quantities. All quantities in this section are 
real, though in practice some of the theory can be extended to complex quantities, as, for example, in the 
theory of time series. 


Definitions: 


Ifw = ly’, z' |" is a random vector, then y and z are said to be uncorrelated if and only if their cross- 
covariance cov(y,z) = 0. We say that y and z are statistically independent if and only if the probability 
density function (pdf) of w is the product of the pdfs of y and z. 
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that all x; have the same covariance matrix & and are uncorrelated, i.e., cov(x,, x; ) = 5,; 2, where 6,, = 1 
forr = s and 6,, = 0 forr 4 s. We call a matrix with the above properties a data matrix. As mentioned 
in the introduction, the x; often constitute a random sample, which we now formally define. A random 
sample of vectors x; (i = 1,2,...,m) of size n consists of n random vectors that are independently and 
identically distributed (i.i.d.), that is, they are statistically independent of each other and have the same 
pdf. When this is the case, we see from the data matrix above that the n elements of x) form a random 
sample from the jth characteristic or variable, being just the jth elements in each x;. However, x‘) and 
x") ( j #k) can be correlated. 
The moment generating function or m.g.f. of any random vector x is defined to be M,(t) = E(e'*). 
It can be used for finding such things as the mean and covariance matrix, or the distribution of related 
random variables. 


| [x x, ...,x] bean n x d matrix of random variables with rows x} such 


Facts: 


1. Forarandom variable x, M,(t) = E(e*’) = ae E(x’) f. The coefficient of 5 in the power series 
expansion of M,(t) gives us E(x") from which we can get 4 = E(x) and var(x) = E(x”) — w?. 
2. Statistical independence implies zero covariance but not generally vice versa except for one notable 


exception, the multivariate normal distribution (see next section). 


53.2 Multivariate Normal Distribution 


Definitions: 


Let x = [xj] = [x1,x2,...,xa]? bead x 1 random vector with mean p and positive definite covariance 
matrix ©. Then x is said to follow a (nonsingular) multivariate normal distribution when its pdf is 


f(x: ps Z) = (20) 4? (det D)7? exp{—F (x — pw) D7/2(x — p)}, 


where —0o < xj < ©, j = 1,2,...,d. We write x ~ Na(u, &). When d = 1 we have the uni- 
variate normal distribution. When » is singular, the pdf of x does not exist but it can be defined via a 
transformation x = Ay where y does have a nonsingular distribution. This is mentioned further below. 

The noncentral y*-distribution with p degrees of freedom and noncentrality parameter 5, denoted by 
Xp, is the distribution of x"x when x ~ Np(w, Ip) and = we" yp. The distribution is said to be central 
whenever 5 = 0, and we denote it by cee 

The noncentral F-distribution with m,n degrees of freedom and noncentrality parameter 5, denoted 
by F (m,n; 5), is the distribution of the so-called F-ratio F = ulm where u~ Naw vr x2 and u and v 


v/n? 


are independent. The distribution is central whenever 5 = 0 and we then write F ~ F (m,n). 


Facts: 


All the following facts except those with a specific reference can be found in [And03, Ch. 2], [Rao73, 
Sec. 8a], or [SLO3, Ch. 1]. 

We can extend our definition to include the singular multivariate normal distribution using one of the 
first two facts below as a definition, each of which includes both the nonsingular and singular cases. When 
we write x ~ Ng(p, 2), we include both possibilities unless otherwise stated. 
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1. Assuming for the trivial case that y ~ N,(b,0) means y = b with probability 1, then the random 
d x 1 vector x follows a multivariate normal distribution if and only if y = a’ xis univariate normal 
for all d x 1 vectors a. 

2. Arandom d x 1 vector x with mean yz and covariance matrix X follows a multivariate normal 
distribution of rank m < d when it has the same distribution as Az + js, where A is any d x m 
matrix satisfying © = AA? andz ~ N,(0, Im). 

3. Given x ~ Nqa(, Z), then the m.g.f. of x is M,(t) = et ett Et When D is positive definite, this 
m.g.f. uniquely determines the nonsingular multivariate normal distribution. 

4. Given x ~ Ng(p, X=) and Am x d, then Ax ~ N,,(Aps, AXA’). The distribution is nonsingular 
if and only if AX has full row rank m(m < d), which is assured when » is positive definite and A 
has rank m. 

5. Any subset of the components of a multivariate normal random variable is multivariate normal. 

6. If the cross-covariance matrix of any two vectors which contain disjoint subsets of x is zero, then 
the two vectors are statistically independent. 

7. If the cross-covariance matrix cov( Ax, Bx) = 0, then Ax and Bx are statistically independent. 

8. If X is positive definite, then (x— ys)? D7!(x— ys) ~ x7, the central x?-distribution with d degrees 
of freedom. 

9. Letx be an x 1 random vector with E(x) = ys and var(x) = ©, and having any distribution. If 
A is any symmetric n x 1 matrix, then E[(x — a)’ A(x — a)] = tr(AZ) + (ws — a)? A(ps— a). 

10. Let x = [x;] be ann x 1 random vector with E(x;) = 6; and uw, = E(x; — 0;)'3 r = 2,3,4. If A is 
any n x n symmetric matrix and a is the column vector of the diagonal elements of A, then 


var(x! Ax) = (4 — 3yu3)a‘a + 2y5tr A? +407 A726 + 430" Aa. 


Examples: 


1 


1. Ifx ~ N, (0,07), then M,(t) = e2? from Fact 3 above. Since pL = 0, uw, = E(x") can be found 
by expanding M,(t) and finding the coefficient of ¢”/r!. For example, uw. = 07,43 = 0, and 
fa= 3p. 

2. If the x; (i = 1,2,...,m) are iid. as N,(0, 07), then by either multiplying pdfs together for 
independent random variables or using m.g.f.s we find that x ~ N,,(0, o7I,)- Substituting the 
results from the previous example into Fact 10 we get var(x?Ax) = 20‘trA?. 


53.3 Inference for the Multivariate Normal 


Definitions: 


Given a random sample {x;} that are i.id., from the nonsingular multivariate normal distribution, the 
likelihood function is defined to be the joint pdf of the sample expressed as a function of the unknown 
parameters, namely, 


L(u,Z) =|] fxs, 2). 


i=1 


The parameter estimates that maximize this function are called the maximum likelihood estimates. The 
sample mean of the sample is defined to be X = )>/_, x;/n, and is not to be confused with the complex 
conjugate. 
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Facts: 
1. The maximum likelihood estimates of 2 and © are, respectively, 47 = xX and == Q/n = 
Soy (xi — &)(x; — x)"/n. Here, “~” denotes “estimate of” in statistics. 
2. Q= X™CX,where X is the data matrix previously defined and the centering matrix C = I,— 4 117 
is symmetric and idempotent. Also, CX = [x, — X,...,X, —x]’ = X, say, and Q= X?X. 


53.4 Principal Component Analysis 


Definitions: 


Let x be a random d-dimensional vector with mean yz and positive definite covariance matrix DU. Let 
T = [t),t,...,tg] be an orthogonal matrix such that T7XT = A = diag(A,,A2,...,Aq), where A, > 
Az >+++ > Ag > Oare the ordered eigenvalues of Xu. The sum trd is sometimes called the total variance. 
Ify = [yj] = T'(x— p), then y= ti (x — pw) (j = 1,2,...,d) is called the jth population principal 
component of x, and z; = Ae : y; is called the jth standardized population principal component. 

In practice, 2 and & are unknown and have to be estimated from a sample x), X2,...,x,. Assuming 
that the underlying distribution is multivariate normal we can use the estimates of the previous section, 
xand © = X7X/n. Carrying out a similar factorization on E as we did for 5, we obtain the eigenvalues 


Ay > Ay = +++ > Ag > Oand an orthogonal matrix T = [t,,t,..., tg] of corresponding eigenvectors. 
For each observation x;, we can define a vector ¥; = T" (x; — x) of sample principal components or 
estimated principal components yielding Y" = [¥,,¥2,...,¥n] = T’X?. 


Although the above method is mainly used descriptively, asymptotic inference for large n can be carried 
out on the assumption that the underlying distribution is normal. 


Facts: 
The following facts can be found in [Chr01, Sec. 3.1—-3.3], [Rao73, Sec. 8g2], or in [Seb04, Sec. 5.2]. 


1. As var(y) = A, which is diagonal, the y; are uncorrelated and var(y;) = Aj. 

2. ae var(yj) = baer var(x;) = tr&. We can use 4;/trx to measure the relative magnitude of 
Aj. Ifthe A; (i = k + 1,...,d) are relatively small so that the corresponding y; are “small” (with 
zero means and small variances), then yu.) = [y1, yas... > Ye] T can be regarded as a k-dimensional 
approximation for y. Thus, yi.) can be used as a “proxy” for x in terms of explaining a major part 
of the total variance. 

3. Let Tix) = [ty,...,t,]. Then: 


* max var(a’x) = var(t} x) = var (t{ (x _ L)) = var(y|) = a1, 
a‘a=1 
so that y; is the normalized linear combination of the elements of x — yz with maximum vari- 
ance A}. 

. max var(a! x) = var(t/ x) = var(yz) = Ax, so that th — p) is the normalized linear 
ala=l,Ty_,a=0 
combination of the elements x — yz uncorrelated with y,, y2,..., yz—-1, With maximum variance 
tes 


4. jij = tj (x; — x), the score of the ith individual on the jth sample principal component, is related 
to the orthogonal projection of x; — X onto range(t; ), namely Pi, (xi —X) = tit? (x; x) Spit 


Examples: 


1. Suppose we assume that each x; is just an observed vector, i.e., a constant. Let v be a discrete random 
vector taking the values x; (i = 1,2,...,) with probability 4, Then E(v) = See 


j-. Xi P(v=x;j) = 
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Soi, xi/n = x and, similarly, var(v) = E. This means that the sample principal components for 
x are the population components for v so that all the optimal properties of population principal 
components hold correspondingly for the sample principal components. 

2. Entomologists are interested in the number of distinct taxa present in a population of winged 
aphids as they are difficult to identify. Forty aphids were trapped and 19 variables on each aphid 
were measured. A principal component analysis was carried out and the first two components 
accounted for 85% of the estimate of the total variance giving an effective reduction from 19 to 
2 dimensions. The 2-dimensional y; were plotted showing the presence of four groups. Also, the 
t, (r = 1,2) suggested which two linear combinations of the 19 measurements gave the best 
discrimination. As the aphids came from slightly different populations, the original assumption 
that the sample is from a homogeneous population is not quite true, but it does provide a starting 
place for further study, e.g., use discriminant coordinates described in the next section with four 
groups. 


53.5 Discriminant Coordinates 


Definitions: 


Suppose we have n d-dimensional observations of which n; belong to group i from the ith underlying 
population (i = 1,2,...,g;n = a n;). Let x;; be the jth observation in group i, and define 


Levey jar ij — X;.)(xj; — X.), the within-groups matrix, and let W, = )7#_, ;(%. — 
%..)(X. — x)’, the between-groups matrix. Since W, and W, are positive definite with probability 1, 
the eigenvalues of Ww," W,, which are the same as those of We? 2Wh w,/ 2\ are positive and distinct with 
probability 1, say, >A, >--+>Aqg> 0. 

Let W, | Woe, = i,¢, be suitably scaled eigenvectors and define C? = [cj,¢,...,cx] (k < d). If 
we define yj; = Cx;;, then the k elements of z;; are called the first k discriminant coordinates (or 
canonical variates). These coordinates are determined so as to emphasize group separation, but with 
decreasing effectiveness so that k has to be determined. The coordinates can be computed using an 
appropriate transformation combined with a principal component analysis. Typically the ¢; are scaled so 
that CSCT = I,, where S = W,/(n — g). 


Examples: 


1. Trivariate measurements x;; were taken on the skulls of six collections (four subspecies with three 
collections apparently from the same species) of anteaters. Using six groups of data from six 
underlying populations in the above theory, we found that [A,,A2,A3] = [2.400, 0.905, 0.052]. 
Since A3 was small, we chose k = 2. We could, therefore, transform the 3-dimensional observations 
x;; into the 2-dimensional y;; = Cx;; and still account for most of the variation in the data. The 
2-dimensional reductions were then plotted to help us look for any patterns. As an aid to our 
search, it was possible to draw a circle with center the reduced mean of each group so that the circle 
contained any reduced random observation from its population with probability 0.95. The center 
of the circle locates the center of gravity of the group and the boundary of the circle shows where 
the bulk of the observations are expected to lie. The closeness of these circles indicated how “close” 
the groups and subspecies were. We found that three of the circles almost overlapped completely, 
confirming a common underlying species, and the remaining circles had little overlap, suggesting 
the presence of four species. 
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53.6 Canonical Correlations and Variates 


Definitions: 
Letz = | denote a d-dimensional random vector with mean yz and positive definite covariance matrix 
y 


&. Let x and y have dimensions d, and d, = d — dj, respectively, and consider the partition 


sys ie a 
oi Liza 
where &;; isd; x d; and X2, = Sy has rank r. Let PR be the maximum value of the squared correlation 
between arbitrary linear combinations a?x and 3"y, and let a = a; and 3 = b; be the corresponding 
maximizing values of a and 2. Then the positive square root / pr is called the first (population) canonical 
correlation between x and y, and u, = alx and vy; = bly are called the first (population) canonical 
variables. 


Let p3 be the maximum value of the squared correlation between a?x and B'y, where @ 
related with a! x and 3’y is uncorrelated with b/y, and let uw. = ax and v. = bly be the maximizing 


Tx is uncor- 


values. Then the positive square root Jp is called the second canonical correlation, and u2 and v2 are 
called the second canonical variables. Continuing in this manner, we obtain r pairs of canonical vari- 
ables u = [u,t,...,uU,]? andv = [v1,v,...,v,]7. We can then regard u and v as lower-dimensional 
“representations” of x and y. 


Facts: 
1. [Seb04, Sec. 5.7] If m = rankXj. > 1 and » is positive definite, then MG ih hat has m 
positive eigenvalues p? > p? > --» > p2, say, and p? < 1. Moreover, Dy;'Xy25' U2) and 
pee ZX») ar X12 have the same (nonzero) eigenvalues. Let a), a2,..., a, and bj, b2,...,b,, be the 


corresponding eigenvectors of ©yq' Ly UV55' Va and Uj Uz, H7]' Dio, respectively. Suppose that 
qa and £ are arbitrary vectors such that for r < m, al 


x is uncorrelated with each aj x G = 
1,2,...,r — 1), and B"y is uncorrelated with each bry (P22 = Dole ajx and 
vj= bry, for j = 1,2,...,r. Then: 


* The maximum squared correlation between ax and "y is given by p? and it occurs when 
a =a, and B= b,. 

* cov(uj, uz) =O for j Ak, and cov(v;, vz) = 0 for j #k. 

* The squared correlation between uj and v; is pi. 

¢ Since po; is independent of scale, we can scale a; and b; such that ay ya; = land bj Xaab,; = 1; 
The u; and v; then have unit variances. 

° Ifthe d; x d) matrix Dy has row full rank, and d, < d, then m = dy. All the eigenvalues of 
Si Xp Die &» are then positive, while 3s dy) = Dy has d; positive eigenvalues and d, — dy 
zero eigenvalues. The rank of Xj) can vary as there may be constraints on X49, such as Yj. = 0 
(rank 0) or Xj. = aly, 1}, (rank 1). 


2. [BriOl, p. 370]. Suppose x and y have means px and py, respectively. Let u = A(x — px) and 
v = B(y — py), where A and B are any matrices, each with r rows that are linearly independent, 
satisfying Ad, A! = I, and BX» BY" = I,. Then E[(u— v)!(u — v)] is minimized when u and v 
are vectors of the canonical variables. 

3. [BS93], [SS80] When » is singular, then the (population) canonical correlations are the positive 
square roots of the eigenvalues of ©), X12) X21; any generalized inverses X{, and X}, may be 
chosen. There will be u canonical correlations equal to 1, where u = rank(,,) + rank(22.) — 
rank(). 
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53.7. Estimation of Correlations and Variates 


Definitions: 
x a 
Let Z),Z2,...,Z, be a random sample and let z = Pe a | be the sample mean and © = 


4 Si, (z; — 2)(z; — 2)", where & is partitioned in the same way as Z, namely 


~ ke a he 
nx = — ; 


VOX5 , We Qa Qzr 
say. Assuming that d) < do, letr? > r? > +--+ > r3. > 0 be the eigenvalues of OF Qn Oo Qo, with 
corresponding eigenvectors 4),4),...,44,. We define uj; = a} (x; — x), the ith element of uj = Xa;, 


where 4; is scaled so that 1 = nar x" Xa; — UA uj;/n — ujuj/n. Then Vr; is the jth sample 
canonical correlation. Moreover, u;; is the jth sample canonical variable of x;. In a similar fashion, we 
define vj; = bi (yi — y), the ith element of v; = Yb j, to be the jth sample canonical variable of y;. Here, 
b,, b2,...,b,g, are the corresponding eigenvectors of Ory Qr1 OF; Qy, scaled so that vjvj/n = 1. The uj; 
and v;; are the scores of the ith observation on the jth canonical variables. 


Facts: 
1. When » is positive definite and n — 1 > d, then, with probability 1, n& is positive definite and 
rankQ), = dy. 
2. The Canonical correlations , [r+ are all distinct with probability 1. 


3.7 ; is the square of the sample correlation between the canonical variables whose values are in the 
vectors uj; and vj. 


Examples: 


1. Length and breadth head measurements were carried out on the first and second sons of 25 families 
to see what relationships existed. Here, x refers to the two measurements on the first son and y to the 
second. It was found that r; = 0.7885 and r. = 0.0537, indicating that just one pair of sample canonical 
variables (u;,v1) would give a reasonable reduction in dimension from 2 to 1. Plotting u; against vj 
using the 21 pairs (uj, vj) gave a reasonably linear plot. The first linear combinations (1), v,) of the two 
measurements suggested that they could be interpreted as a measure of “girth” while the second (uz, v2) 
could be interpreted as “shape” measurements. We can conclude that there is a strong correlation between 
the head sizes of first and second brothers, but not between the head shapes. 


53.8 Matrix Quadratic Forms 


To carry out inference for multivariate normal distributions we now require further theory. 


Definitions: 


If X is ann x d data matrix and A = [a;;] is a symmetric n x n matrix, then the expression X TAX = 
pa ae aj jxix} is said to be a matrix quadratic form. We have already had one example, namely, 
Q = X'CX in section 53.3. When the {x;} are a random sample from a distribution with mean ps and 
covariance matrix & we define S = Q/(n — 1) to be the sample covariance matrix. (Some authors define 
it as © in Section 53.3) 

Ifthe x; arei.id.asNq(0, X&),withd < nand X = [o;;] positive semidefinite, then W = X' X is saidto 
a have a Wishart distribution with n degrees of freedom and scale matrix 2, and we write W ~ W,(n, x). 
If & is positive definite then the distribution is said to be nonsingular. 
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Suppose y ~ Nq(0, =) independently of W ~ Wa(m, X) and that both distributions are nonsingular. 
Then Hotelling’s T’ distribution is defined to be the distribution of T? = my’ W~'y and we denote this 
distribution by T? ~ T7,,. 

Finally, in inference we are usually interested in contrasts a! p, in the z;, where a is a vector whose 
entries sum to 0, ie., a’ 1g = 0. Examples are 4, — 2 = [1,—1,0,..., 0] pr and py — + (2 + 3). 


Facts: 
All facts, unless otherwise indicated, appear in [Seb04, Secs. 2.3—2.4, 3.3-3.4, 3.6.2]. 


1. E(S) = &, irrespective of the distribution of the x;. 
2. [HS79] Let X be an n x d data matrix and let A and B be matrices of appropriate sizes. If vec is 
the usual “stacking” operator, then: 
* cov[vec (AX B), vec(CXD)] = (B'XD) @ (AC"). 
° If U = AXB and V = CXD, then U and V are pairwise uncorrelated, that is, cov(ujj,v-s) = 0 
for alli, j,r, ands, if AC’ = 0 and/or B'XD = 0. 


3. Suppose W = [wj;;] has a Wishart distribution W4(m, X), with & possibly singular. Then the 
following hold: 
° E(W) = md. 
* Let Abeagq x d matrix with rank q < d. Then AWA’ ~ W,(m, AXA’). When & is positive 
definite, then this distribution is nonsingular. 
* When oj; > 0, then wjj/ojj ~ x23 (j = 1,2,...,d). However, the w;; (7 = 1,...,d) are not 
statistically independent. 
* When » is positive definite, then det & > 0 and det W/ det ¥ is distributed as the product of d 
independent chi-square variables with respective degrees of freedom m,m — 1,...,.m—d+1. 
4. (See [SK79, p. 97], [Sty89].) Let W ~ Wa(m, &) with & possibly singular, and let A be ann x n 
symmetric matrix. Then: 
* E(WAW) = m[ZT ATE + tr(AL)U] + nP DAL. 
° Ifm > d+ 1and & is nonsingular, then 


1 
E(WAW7!) = ————_[mAz7! — A? — (trA)I], 
m—d—1 


1 
E(W7!AW) = mat Az — A’ — (trA)I]. 
° Ifm > d+3 and & is nonsingular, then 


(m—d—2)D7 Ad! 4+ D1 ATSO! — (trAdo)=E7! 
(m — d)(m — d — 1)(m—d — 3) 


E(W AW") = 


5. Suppose that the rows x} of X are uncorrelated with var(x;) = %; fori = 1,2,...,n, and A isan 
n X n symmetric matrix. Then: 
* E(X7AX) = S° aii Xj + E(X")AE(X). 
i=l 


° If ©; = & for alli, then E(X7 AX) = (trA)D + E(X") AE(X). 


6. [Das71, Th. 5], [EP73, Th. 2.3] Suppose that the rows of X are independent and A is ann x n 
positive semidefinite matrix of rank r > d. If for each x; and all b and c with b ¥ 0 the probability 
Pr(b’x; = c) = 0, then X7 AX is positive definite with probability 1. 
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7. Suppose that rows of X are iid. as Ny(0, X), with © possibly singular, and let A and B ben x n 

symmetric matrices. 

* If Ahas rankr < d, then X?AX ~ Wa(r, X) if and only if A= A’. 

* Suppose X7 AX and X7 BX have Wishart distributions. They are statistically independent if and 
only if AB = 0. 

8. Referring to the definition of Hotelling’s T’, maT? ~ F(d,m— d +1), the F-distribution 
with d and m — d + 1 degrees of freedom. When y ~ Ng(9, 2), then F ~ F(d,m—d-+ 1;8), the 
corresponding noncentral F-distribution with noncentrality parameter 6 = 07X80. 

9. Suppose that the rows of X are 1.i.d. as nonsingular Ng(, =). Then: 

* x&~ Na(y, U/n). 
* Q=(n—-1)S ~ Wa(n— 1, 2%). 
* xand S are statistically independent. 
° T? =n(x— p)'S“"(k— pp) ~ Tr This statistic can be used for testing the null hypothesis 
Ao : = bo. 
10. Given Hy : pt € VY, where VY is a p-dimensional vector subspace of IR4, then: 


Dos 2 72 
. Tin = MiNpey T Te pnts 


° If Hp : w = K{, where K is a known d x p matrix of rank p and @ is a vector of p 
unknown parameters, then YV = range(K), and Te = n(x’ S~'x — x™S-!K*), where 
PF S(KS Kr hs? se 
11. If Hp : Aw = 0, where A is (d — p) x d of rank d — p, then V = ker(A) (also called the null space) 
and T?,, = n(Ax)?(AS A?) Ax. 
* Let Abeag x d matrix ofrankq < d. Then the quadratic n(Ax— Ay)? (AS.A™)—!(Ax— Ap) ~ 
Te This can be used for testing Hy : A = c. 


¢ If A is a matrix with rows which are contrasts so that Alz = 0, then 


n(x? S-} ly)’ 


nx" AT(AS AT)! Ax = nx" S7'x — —— 
1; S11 


12. Let vj, V2,...,V,, bea random sample from Nq(js1, 2) and let wi, W2,...,;Wn, be an independent 

random sample from Ng(f2, =), both distributions being nonsingular. Let Q; = Se - 
v)(v; —¥)? and Q, = ye Oy — w)(w; — w)?. If @ = a, — po and Aisagq x d matrix of rank 
q < d, then: 
°Z2=0—w~Nu(0,(2+2)2). 
* Q=Qi4t+ Q ~ Wal + m — 2,2). 
¢ The statistic 

n\n 


ny + Ny 


(AZ= BO) (AS yA?) AZ AG) Te |. 


where S, = Q/(n, + nz — 2). This can be used to test Hy : A@ = 0. When A is a certain 
(d — 1) x d contrast matrix, then the methodology relating to Hp is called profile analysis. 


Examples: 


1. Suppose the rows of X are i.i.d. nonsingular Ny(0,£). Then = X71,,/n and W, = nxx? = 
XT AX, where A = 1,17 /n is symmetric and idempotent of rank 1. Also, from Section 53.3, 
Wy = (x; —x)(xj —x)? = X7CX, where C = I, — Ais symmetric and idempotent of rank n — 1. 
Since AC = 0, we have from Fact 7 above that W;) ~ Wq(1, 2), W2 ~ Wa(n — 1, %), and W, and 
W, are statistically independent. 


Multivariate Statistical Analysis 53-11 


2. (cf. Fact 10) Suppose we have a group of 30 animals all subject to the same conditions, and the 
length of each animal is observed at d points in time ft), h,..., ta. The d lengths for the ith animal 
will give us a d-dimensional vector, x; say, and we assume that the x; are all i.i.d., as nonsingular 
Na(pt, Z). We are interested in testing the hypothesis Hp that the “growth curve” is a third degree 
polynomial so that Hp : 4; = Bo + Bit; + Bot} + Bst} (j =1,2,...,d) or Hy: w= KB, where 
the jth row of K is [1,¢;, ti, t}] and 3 = [Bo, Bi, B2, B3]" is a vector of unknown parameters. 
We then test Hy by calculating T;?;,, given in Fact 10, and this statistic, which has a Hotellings T* 
distribution when H is true, can then be converted to an F -statistic using Fact 8. If this F -value is 
significantly large we reject Hp. 

3. (cf. Fact 11) The lengths of the femur and humerus on the left- and right-hand sides are measured 
for n = 100 males aged over 40 years to see if men are symmetrical with respect to the lengths of 
these bones. For the ith male, we therefore have a four-dimensional observation vector x;. Assuming 
that the population of measurements is multivariate normal with mean pp = [[11, [L2, (43, a]? 

where 4; and /2 refer to the left side, then we are interested in testing the contrasts 4; — “3 = 0 

and jl2 — 4 = 0. Thus, A has rows [1,0, —1, 0] and [0, 1,0, —1], giving us Hp : Al, = 0. We can 

then test Hp using Te defined in Fact 11, and this T? statistic is then converted to an F -statistic, 


as in the previous example. 


> 


53.9 Multivariate Linear Model: Least Squares Estimation 


Definitions: 


Let Y be ann x d data matrix which comes from an experimental design giving rise to n observations, the 
rows of Y. Then we can use observations on the jth characteristic (variable) y\!’, the jth column of Y, to 
construct a linear regression model y) = 94 +u = KB” + u as in Section 52.2 (with K replaced 
by X there). Because the design is the same for each variable, the design matrix K will be independent of 
j (j = 1,2,...,d), though the models will not be independent as the y‘/) are not independent. Putting 
all d regression models together, we get Y = © + U, where © = KB, B is p x d matrix of unknown 
parameters, K isn x p ofrankr (r < p),andU = [u",... ju} = [uy,...,u,]". We shall assume 
that the u; are a random sample from a distribution with mean 0 and covariance matrix © = [o;;]. Then 
Y = KB + U is called a multivariate linear model. When d = 1, this reduces to the univariate linear 
model or Gauss—Markov model; see Chapter 52 and [Seb04, ch. 8]. 


Now, for the jth model, 6” = Py‘) is the (ordinary) least squares estimate of 9), where P = 
K(K'K)-K’, and © = PY is defined to be the least squares estimate of ©. When r = p, then setting 
© = KB we have B = (K?K)~'K?6 = (K’K)~!K‘Y, called the the least squares estimate of B. If 
r < p,then B isnot unique and is given by B =(K'K)-K‘Y, where (K?K)~ is any generalized inverse 
of KTK. 

If K has less than full rank, then each of the d associated univariate models also has less than full rank. 
Moreover, al 3 is estimable for eachi = 1,2,...,q and each model j = 1,2,...,d ifaj € range(K"), 
Let A = [aj,a),.. agi. Combining these linear combinations we say that AB is estimable when 
A= LK forsomeq x n matrix L. 


Facts: 
All the following facts can be found in [Seb04, Ch. 8]. 
1. If K? = [k,,...,k,] then, transposing the model, y; = B?k; + uj. 
2. Thecross-covariance cov(y;,y;) = cov(u,,u;) = 6,; ,whered,, = 1whenr = s and 0, otherwise. 
3. The cross-covariance cov(y3), y) = coviu'),u) = ojklq for all j,k =1,...,d. 
4. If K has full rank p, then 8”) = (KTK)“1KTy and cov(B'”, 8") = oj4(K™K)~? (all 
jk =1,...,d). 
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5. Let F(@) = (Y — O)"(Y — ©). Then: 


* F(®)=Y7(1, — P)¥ =U" (I, — P)U. 
* E[U'(I, — P)U] =(n—r)>. 


6. F(®) — F(O) is positive semidefinite for all © = XB, and equal to 0 ifand only if 0 = ©. We say 
that © is the minimum of F(Q). Then: 


* trF(©) > trF(6). 
* det F(@) > det F(O). 
* ||F(©)|| = ||F(©)||, where || Al] = {tr(AAT)}17. 
Any of these three results could be used as a definition of 6. 
7. Multivariate Gauss—Markov Theorem. If @ = a hj" ) a linear combination of all the elements 


of ©, then d — as : hr” is the linear unbiased estimate with minimum variance or BLUE (best 


linear unbiased estimate) of @. (See also Chapter 52.) 


Examples: 

4 
wP 
we see that the two-sample problem mentioned in Fact 12 of Section 53.8 is a special case of the 
multivariate model with 


1. [Seb04, Sec. 8.6.4] By setting vi= [V1,V2,---5 Vn] W! = [w,w,. +5 Wn, ], and Y = 


53.10 Multivariate Linear Model: Statistical Inference 


Definitions: 


Let Y = © + U, where © = KB, be a multivariate linear model. We now assume that the underlying 
distribution (of the u;) is a nonsingular multivariate normal distribution N4(0, =). 


Facts: 
The following facts appear in [Seb04, Sec. 8.6]. 


1. The likelihood function for Y, i-e., the pdf of vec Y, can be expressed in the form 
1 
(27 )~"4/? (det 5)? expttr[—3(¥ —@)'x"“l(Y — ©)]}. 


2. The maximum likelihood estimates of © and @ that maximize the likelihood function are © 
(the least squares estimate) and © = E/n, where E = (Y — ©)7(Y — ©); E is usually called 
the residual matrix or error matrix. If the n x p matrix K has full rank (i.e., rank p), then the 
maximum likelihood estimate of B is B. 

3. Let E be the residual matrix and assume n — r > d. Then 
* E is positive definite with probability 1. 
°* E~W,(n-—1,X). 

* E is statistically independent of 6, and of B if K has full rank. 


* The maximum value of the likelihood function is (277)~"4/2(det £)-"/2e-"4/2, 


If K has full rank, then 8” ~ N, (8 ,0);(K7K)-}). 
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4. Suppose that K has rankr < p. Let Abeaknownq x p matrix of rank qg and let AB be estimable. 
We are interested in testing Hy : AB = C, where C is known. 


* The minimum, E y say, of (Y — K B)?(Y — KB) subject to AB = C occurs when B equals 
B= B—(KR) A [AKIRA AB =O 
Although Ba is not unique, Ox = KBr is unique. Moreover, Ey = (Y — On)T(Y _ On) is 
positive definite with probability 1. 

* H= Ey —E =(AB—C)"[A(K?K)-A‘]-!(AB — C) is positive definite with probability 1, 
and H and E are statistically independent (both irrespective of whether Hp is true or not). 

* E(H) = q= + (AB — C)"[A(X?X)-A?]-!(AB — C) = qu + D, where D is positive def- 
inite; D is zero when Hp is true. This means that E(H), and, therefore, H itself, tends to be 
“inflated” when Hp is false so that a test statistic for Hp can be based on a suitable function 
of H. 

¢ When Fh is true, H ~ Wa(q, X). 

* Let EP? be the positive definite square root of E 7. Then when Hp is true, V = E os "HE a af a 


scaled function of H, has a d-dimensional multivariate Beta distribution with parameters q and 
nf. 


5. [Seb04, Sec. 8.6.2] Four different criteria are usually computed for testing Ho, and these are essen- 
tially based on the eigenvalues of V given above, which measure the scaled magnitude of H in some 
sense. They detect different kinds of departures from Hp and are all significantly large when Hp is 
false. 

* Roy’s maximum root statistic max, the maximum eigenvalue of HE~!. 
* Wilks’ Lambda or likelihood ratio statistic A = (det E / det Ey)". 

* Lawley—Hotelling trace statistic (n —r)tr(HE™'). 

¢ Pillai’s trace statistic tr(HE ae 


Example: 


1. [Seb04, Sec. 8.7.2] To test the general linear hypothesis Hp : ABD = 0, where A is q x p of rank 
q < pand Disd x v of rank v < d, we let Yp = YD so that the linear model Y = KB + U is 
transformed to 


Yp=KBD+UD=KA+U), 


say, where the rows of Up are i.i.d. N,(0, D™ZXD). Then Hy becomes AA = 0 and 


* Hbecomes Hp = D'HD = (ABD)"[A(K?K)-!A™]-! AB Dand E becomes Ep = D'ED ~ 
W,(n—1r, D' XD). 


* When ABD = Ois true, then Hp ~ W,(q, D?XD). 


We can now use Facts 4 and 5 above to test Hp using Hp and E p instead of H and E. This hypothesis 
arises in carrying out a so-called profile analysis of more than two populations. 


53.11 Metric Multidimensional Scaling 


Definitions: 


Given a set of n objects, a proximity measure d,, is a measure of the “closeness” of objects r and s; here, 
“closeness” does not necessarily refer to physical distance. We shall consider only one such measure as 
there are several. A proximity d,, is called a (symmetric) dissimilarity if d,, = 0, d,; > 0,andd,, = d,,, 
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for allr,s = 1,2,...,n; the matrix D = [d,.] is called a dissimilarity matrix. We say that D is Euclidean 
if there exists a p-dimensional configuration of points y;, y2,...,Yn for some p such that all interpoint 
Euclidean distances satisfy ||y, — y;|| = d,s. 


Facts: 


1. Let A = [a;;] be asymmetric n x n matrix, where a,, = — id. Define b,, = a;; — ay. — 4.5 +@.., 
where @,. is the average of the elements of A in the rth row, @.; is the average for the sth column, 
and @.. is the average of all the elements. Hence, B = [b,,] = CAC, where C = I — +117 is the 
usual centering matrix. Then D = [d,,] is Euclidean if and only if B is positive semidefinite; see 
[Seb04, p. 236]. 

2. If D is not Euclidean, then some eigenvalues of B will be negative. However, if the first k eigenvalues 
are comparatively large and positive, and the remaining positive or negative eigenvalues are near 


zero, then the rows of Y, = [y', y?),...,y] will give a reasonable configuration. If the original 
objects are d-dimensional points x; (i = 1,2,...,m) to begin with so that ||x, — x, | = d?., then 
D is Euclidean and the n rows of Y; will give a k-dimensional reduction of a d-dimensional system 
of points. 

Example: 


1. The migration pattern that occurred with the colonization of islands in the Pacific Ocean can be 
investigated linguistically and values of d,,; can be constructed based on linguistic information. For 
example, the proportion p,; of the words for, say, 50 items that islands r and s have in common 
can be used as a measure of similarity (with p,, = 1), which we can convert into a dissimilarity 
d,s = (2 — 2p;s)!/?. Then a,; = —}d?, = p,; — 1 and, when computing b,;, the —1 drops out 
so that we can leave it out and set a,, = p,;. If A is positive semidefinite then so is B, which 
implies that D is Euclidean. Using B, we can then find a Y; from which we can construct a lower 
dimensional map to see which islands are closest together linguistically. The same method can also 


be applied to blood groups using a different d,,. 
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Markov chains are encountered in several applications arising in different contexts, and model many real 
problems which evolve in time. Throughout, we denote by PLX = j] the probability that the random 
variable X takes the value j, and by P[X = j|Y = i] the conditional probability that X takes the value 
j» given that the random variable Y takes the value 7. Moreover, we denote by E[X] the expected value of 
the random variable X and by E[X|A] the conditional expectation of X, given the event A. 


54.1 Basic Concepts 


Definitions: 


Given a denumerable set E , a discrete stochastic process on E isa family {X,; : t € T} ofrandom variables 
X, indexed by some denumerable set T’ and with values in E, i.e., X; € E for all t € T. Here, E is the 
state space, and T is the time space. 

The discrete stochastic process {X,, : n € N} is a Markov chain if 


P[Xn41 = In+i|Xo = Jo» Xy = Jo» oe Xn = jn] = P[Xn44 = Inti lXn = jnls (54.1) 


for any (n + 2)-tuple of states {jo,...5 jnti} € E, and for all time n € N. 
A Markov chain {X,, : n € N} is homogeneous if 


P[Xn41 = j|Xn = 1] = P[X1 = j|Xo = 7), (54.2) 


for all states i, j € E and for all timen € N. 
Given a homogeneous Markov chain {X,, : n € N} we define the transition matrix of the Markov chain 
to be the matrix P = [p;,j]i,jez such that 


Pig = PLM = j|Xo = 1), for alli, j in E. 


Throughout, unless differently specified, for the sake of notational simplicity we will indicate with the 
term Markov chain a homogeneous Markov chain. 
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A finite Markov chain is a Markov chain with finite space state E; an infinite Markov chain is a Markov 
chain with infinite space state E. 
A row vector 7 = (71;)jeg such that 


T7P=T (54.3) 


is an invariant vector. 
If the invariant vector 7 is such that 


a; > Oforalli, and bie = 1, (54.4) 


i€E 


then 7 is an invariant probability vector, or stationary distribution. The definition of stationary distri- 
bution extends the definition given in Section 9.4 to the case of an infinite matrix P. 


Facts: 


1. The Markov property in Equation (54.1) means that the state X,, of the system at time n is sufficient 
to determine which state might be occupied at time m + 1, and the past history Xo, X1,..., Xn—1 
does not influence the future state at time n + 1. 

2. Ina homogeneous Markov chain, the property in Equation (54.2) means that the laws which govern 
the evolution of the system are independent of the time n; therefore, the evolution of the Markov 
chain is ruled by the transition matrix P, whose (i, j th entry represents the probability to change 
from state i to state j in one time unit. 

3. The number of rows and columns of the transition matrix P is equal to the cardinality of E. In 
particular, if the set E is finite, P is a finite matrix (see Examples 1 and 5); if the set E is infinite, 
P is an infinite matrix, i.e., a matrix with an infinite number of rows and columns (see Examples 
2-4), 

4, The matrix P is row stochastic, i.e., it is a matrix with nonnegative entries such that Se cx Pi,j =1 
for anyi € E,i.e., the sum of the entries on each row is equal to 1. 

5. If |E| < oo, the matrix P has spectral radius 1. Moreover, the vector 1)z) is a right eigenvector 
of P corresponding to the eigenvalue 1; any nonzero invariant vector is a left eigenvector of P 
corresponding to the eigenvalue 1; the invariant probability vector 7 is a nonnegative left eigenvector 
of P corresponding to the eigenvalue 1, normalized so that 71,7; = 1. 

6. Inthe analysis of Markov chains, we may encounter matrix products A = BC,where B = [bj,jJijcz 
and C = [c;,j]i,jez are nonnegative matrices such that jee bij < land jee ¢i,j < 1 for any 
i € E (see, for instance, Fact 7 below). The (i, j)th entry of A, given by aj; = yee BinCh,j> 
is well defined also if the set E is infinite, since 0 < )oy ep binenjy < Sone Vin < 1. More- 
over, A is a nonnegative matrix such that Dick aj; < 1 for anyi ¢€ E. Indeed, Dick a4; = 
jee ek Vache pee Dy jee Chi < doner Vinh < 1. Similarly, the product vy = 
uB, where u = (uj)icg is a nonnegative row vector such that baer u; < 1, is well defined 
also in the case where E is infinite; moreover, v = (v;)jcz is a nonnegative vector such that 
Dieg Vi <1. 

7. [Nor99, Theorem 1.1.3] The dynamic behavior of a homogeneous Markov chain is completely 
characterized by the transition matrix P: 

PX = J|Xa =a) = (Pe, 
for all times n > 0, all intervals of time k > 0, and all pairs of states i and j in E. 

8. In addition to the system dynamics, one must choose the starting point Xo. Let nO = (eye E 
be a probability distribution on E, i.e., a nonnegative row vector such that the sum of its compo- 
nents is equal to one. Assume that eid = P[Xy = i], and define the row vector 7” = (ick 
to be the probability vector of the Markov chain at time n > 1, that is, aie = P[X, = i|Xo]. 
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9. 


10. 


Then 


ert) — 7 p, n> 0, (54.5) 


a) = 7 pr n> 0. (54.6) 


If the initial distribution # coincides with the invariant probability vector 7, then aw”) = mw for 
any n > 0. 

In certain applications (see for instance, Example 5) we are interested in the asymptotic behavior 
of the Markov chain. In particular, we would like to compute, if it exists, the vector limy— 0 wl"), 
From Equation (54.5) of Fact 8 one deduces that, if such limit exists, it coincides with the invariant 
probability vector, ie., limp oo nr = 7, 


Examples: 


Ls 


Random walk on {0,1,...,k}: Consider a particle which moves on the interval [0, k] in unit steps 
at integer instants of time; let X, € {0,1,...,k}, m > 0, be the position of the particle at time 
n and let 0 < p < 1. Assume that, if the particle is in the open interval (0, k), at the next unit 
time it will move to the right with probability p and to the left with probability q = 1 — p; if the 
particle is in position 0, it will move to the right with probability 1; if the particle is in position k, 
it will move to the left with probability 1. Clearly, the discrete stochastic process {X, :n € N} isa 
homogeneous Markov chain with space state the set E = {0,1,...,k}. The transition matrix is the 
(k +1) x (k +1) matrix P = [pj,;]i,j=0,..,z. given by 


From Equation (54.6) of Fact 8 one has that, if the particle is in position 0 at time 0, the probability 
vector at time n = 10 is the vector 


eae ae Sho wore. 


. Random walk on N: If we allow the particle to move on N, we have a homogeneous Markov chain 


with state space the set E = N. The transition matrix is semi-infinite and is given by 
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3. Random walk on Z: If we allow the particle to move on Z, we still have a homogeneous Markov 
chain, with state space the set E = Z. The transition matrix is bi-infinite and is given by 


0 p oO 
= q O p 
0 gq oO 


4. A simple queueing system [BLM05, Example 1.3]: Simple queues consist of one server which attends 
to one customer at a time, in order of their arrivals. We assume that time is discretized into intervals 
of length one, that a random number of customers join the system during each interval, that 
customers do not leave the queue, and that the server removes one customer from the queue at the 
end of each interval, if there is any. Defining a, as the number of new arrivals during the interval 
[n — 1,n) and X,, as the number of customers in the system at time n, we have 


x Xn + On41 —1 if X;+Qn41 = 1 
sa aey pl if X, + @n41 = 0. 


If {a,,} is a collection of independent random variables, then X,,41 is conditionally independent 
of Xo, ..-, Xn—1 if X, is known. If, in addition, the a,,’s are identically distributed, then {X,,} is 
homogeneous. The state space is N and the transition matrix is 


fo+4 42 493 4a 


40 41 42 4B 
P= 


90 41 Q2 
0 


where q; is the probability P[w = i] that new customers join the queue during a unit time interval, 
a denoting any of the identically distributed random variables a,,. Markov chains having transition 
matrix of the form 


B, By Bz Bg 


Ag Ay Az Az 
P= 
Ag Ay Ag 


0 


where Aj, Bj+;,7 > 0, are nonnegative k x k matrices, are called M/G/1-type Markov chains, and 
model a large variety of queuing problems. (See [Neu89], [LR99], and [BLM05].) 

5. Search engines (see Section 63.5, [PBM99], [ANT02] and [Mol04, Section 2.11]): PageRank is used 
by Google to sort, in order of relevance, the pages on the Web that match the query of the user. 
From the Web page Google Technology at http: //www.google.com/technology/: “The 
heart of our software is PageRank’™, a system for ranking Web pages developed by our founders 
Larry Page and Sergey Brin at Stanford University. And while we have dozens of engineers working 
to improve every aspect of Google on a daily basis, PageRank continues to provide the basis for all 


Markov Chains 54-5 


of our Web search tools.” Surfing on the Web is seen as a random walk, where either one starts from 
a Web page and goes from one page to the next page by randomly following a link (if any), or one 
simply chooses a random page from the Web. Let E be the set of Web pages that can be reached by 
following a sequence of hyperlinks starting from a given Web page. If k is the number of Web pages, 
ie., k = |E|, the connectivity matrix is defined as the k x k matrix G = [g;,;] such that g;,; = 1 if 
there is a hyperlink from page i to page j, and zero otherwise. Let rj = Par i,j andc; = ae gj,i be 
the row and column sums of G; the quantities r; and c; are called the out-degree and the in-degree, 
respectively, of the page i. Let0 < q < landlet P = [pj;,;] be the k x k stochastic matrix such that 
Pig = 98i,j/Ti +1. —@)/k,i, 7 =1,...,k. The value q is the probability that the random walk on 
E follows a link and, therefore, 1 — q is the probability that an arbitrary page is chosen. The matrix 
P is the transition matrix of the Markov chain that models the random walk on E. The importance 
of a Web page is related to the probability to reach such page during this randow walk as the time 
tends to infinity. Therefore, Google’s PageRank is determined by the invariant probability vector 7 
of P: The larger the value of an entry 7; of zr, the higher the relevance of the ith Web page in the 
set E. 


54.2 Irreducible Classes 


Some of the Definitions and Facts given in this section extend the corresponding Definitions and Facts 
of Chapter 9 and Chapter 29. Indeed, in these chapters, it is assumed that the matrices have a finite size, 
while in the framework of Markov chains we may encounter infinite matrices. 


Definitions: 


The transition graph of a Markov chain with transition matrix P is the digraph of P, ['(P). That is, the 
digraph defined as follows: To each state in E there corresponds a vertex of the digraph and one defines a 
directed arc from vertex i to vertex j for each pair of states such that p;,; > 0. More information about 
digraphs can be found in Chapter 9 and Chapter 29. 

A closed walk in a digraph is a walk in which the first vertex equals the last vertex. 

State i leads to j (ori has access to j) if there is a walk from i to j in the transition graph. States 7 and 
j communicate if i leads to j and j leads toi. 

A Markov chain is called irreducible if the transition graph is strongly connected, i.e., if all the states 
communicate. A Markov chain is called reducible if it is not irreducible. 

The strongly connected components of the transition graph are the communicating classes of states, 
or of the Markov chain. Communicating classes are also called access equivalence classes or irreducible 
classes. 

A communicating class C is a final class if for every state i in C, there is no state j outside of C such 
that 7 leads to 7. If, on the contrary, there is a state in C that leads to some state outside of C, the class is 
a passage class. A single state that forms a final class by itself is absorbing. In Section 9.4, passage classes 
are called transient classes, and final classes are called ergodic classes; in fact, transient and ergodic states 
of Markov chains will be introduced in Section 54.3, and for finite Markov chains the states in a passage 
class are transient, and the states in a final class are ergodic, cf. Section 54.4. 

A state i is periodic with period 5 > 2 ifall closed walks through 7 in the transition graph have a length 
that is a multiple of 5. A state i is aperiodic if it is not periodic. 

A Markov chain is periodic with period 64 if all states are periodic and have the same period 6. 


Facts: 


1. A Markov chain is irreducible if and only if the transition matrix P is irreducible, ie., if P is not 
permutation similar to a block triangular matrix, cf. Section 9.2, Section 27.1. 

2. Ifwe adopt the convention that each state communicates with itself, then the relation communicates 
is an equivalence relation and the communicating classes are the equivalence classes of this relation, 
cf. Section 9.1. 
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A Markov chain is irreducible if and only if the states form one single communicating class, cf. 
Section 9.1, Section 9.2. 

If a Markov chain with transition matrix P has K > 2 communicating classes, denoted by C,, C>, 
...) Cx, then the states may be permuted so that the transition matrix P’ = TIP TI? associated 
with the permuted states is block triangular: 


lag Pip awed Pix ] 


PK-1K 
0 Prox 


where P;,; is the submatrix of transition probabilities from the states of C; to C;, the diagonal blocks 
are irreducible square matrices, and IT is the permutation matrix associated with the rearrangement, 
cf. Section 9.2. 

[Cin75, Theorem (3.16), Chap. 5] Periodicity is a class property and all states in a communicating 
class have the same period. Thus, for irreducible Markov chains, either all states are aperiodic, or 
all have the same period 5, which we may call the period of the Markov chain itself. 


Examples: 


bs 


2; 


Figure 54.1 is the transition graph associated with the Markov chain on E = {1, 2, 3} with transition 
matrix 


Ie wl ee 
NIF we © 
Bl wl © 


The two sets C; = {1} and C, = {2, 3} are two communicating classes. C2 is a passage class, while 
C is a final class, therefore state 1 is absorbing. 

Figure 54.2 is the transition graph associated with the Markov chain on E = {k €N:k > 1} with 
transition matrix having following structure: 


hi * "| 
0 O- x 
* 0 OO x 
0 0 0 Ox 
p=|9 0 *« O O * 


FIGURE 54.1 Transition graph of the Markov chain of Example 1. 
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FIGURE 54.2 Graph of an infinite irreducible periodic Markov chain of period 3. 


where “x” denotes a nonzero element. This is an example of a periodic irreducible Markov chain 
of period 3. 


54.3 Classification of the States 


Definitions: 


Let T; be the time of the first visit to j, without taking into account the state at time 0, i.e., 
Tj =min{n > 1: X, = j}. 

Define 
fj = PLT} < 00|Xo = j]; 


that is, f; is the probability that, starting from j, the Markov chain returns to j in a finite time. 

A state j € E is transient if f; < 1. 

A state j € E is recurrent if f; = 1. A recurrent state j € E is positive recurrent if the expected 
return time E[T;|Xo = j] is finite; it is null recurrent if the expected return time E[T;|Xo = j] is infinite. 
Positive recurrent states are also called ergodic. 

A Markov chain is positive/null recurrent or transient if all its states are positive/null recurrent or 
transient, respectively. 

A regular Markov chain is a positive recurrent Markov chain that is aperiodic. 

The matrix R = )~~, P” is the potential matrix of the Markov chain. 


Facts: 


1. Transient states may be visited only a finite number of times by the Markov chain. On the contrary, 
once the Markov chain has visited one recurrent state, it will return to it over and over again; if j 
is null recurrent the expected time between two successive visits to j is infinite. 

2. [Cin75, Cor. (2.13), Chap. 5] For the potential matrix R = [r;,;] one has rj,; = 1/(1 — fj), where 
we set 1/0 = co. 

3. [Cin75, Sec. 3, Chap. 5] The (j, j)th entry of R is the expected number of returns of the Markov 
chain to state j. A state j is recurrent if and only if r;,; = oo. A state j is transient if and only if 
r is < ©&. 

4. [Nor99, Theorems 1.5.4, 1.5.5, 1.7.7] The nature of a state is a class property. More specifically, the 
states in a passage class are transient; in a final class the states are either all positive recurrent, or all 
null recurrent, or all transient. 

5. [Nor99, Theorem 1.5.6] Ifa final class contains a finite number of states only, then all its states are 
positive recurrent. 

6. From Facts 4 and 5 one has that, for a communicating class with a finite number of states, either 
all the states are transient or all the states are positive recurrent. 

7. From Fact 4 above, if a Markov chain is irreducible, the states are either all positive recurrent, or all 
null recurrent, or all transient. Therefore, an irreducible Markov chain is either positive recurrent 
or null recurrent or transient. 
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[Nor99, Secs. 1.7 and 1.8] Assume that the Markov chain is irreducible. The Markov chain is positive 
recurrent ifand only if there exists a strictly positive invariant probability vector, that is, a row vector 
a = (7;)icg such that 7; > 0 that satisfies Equations (54.3), (54.4) of Section 54.1. The invariant 
vector 7 is unique among nonnegative vectors, up to a multiplicative constant. 

[Nor99, Secs. 1.7 and 1.8], [BLM05, Sec. 1.5] If the Markov chain is irreducible and null recurrent, 
there exists a strictly positive invariant vector, unique up to a multiplicative constant, such that 
the sum of its elements is not finite. Thus, there always exists an invariant vector for the transition 
matrix of a recurrent Markov chain. Some transient Markov chains also have an invariant vector 
(with infinite sum of the entries, like in the null recurrent case) but some do not. 

[Cin75, Theorem (3.2), Chap. 5], [Nor99, Secs. 1.7 and 1.8] If 7 is a transient or null recurrent 
state, then for anyi € E, lim,_...(P")i,; = 0. If j is a positive recurrent and aperiodic state, then 
limp—oo(P");,; > 0. If j is periodic with period 4, then limti-veatP™) 5 > 0. 

[Nor99, Secs. 1.7 and 1.8] Assume that the Markov chain is irreducible, aperiodic, and positive 
recurrent. Then lim,_,..(P")i,; = 2; > 0 for all j, independently of i, where 7 = (7; )jcz is the 
stationary distribution. 

Facts 3, 4, 6, and 10 provide a criterium to classify the states. First identify the communicating classes. 
If a communicating class contains a finite number of states, the states are all positive recurrent if 
the communicating class is final; the states are all transient if the communicating class is a passage 
class. If a communicating class has infinitely many states, we apply Fact 3 to determine if they are 
recurrent or transient; for recurrent states we use Fact 10 to determine if they are null or positive 
recurrent. 


Examples: 


1, 


Let P be the transition matrix of Example 1 of Section 54.2. Observe that with positive probability 
the Markov chain moves from state 2 or 3 to state 1, and when the Markov chain will be in state 
1, it will remain there forever. Indeed, according to Facts 4 and 5, states 2 and 3 are transient since 
they belong to a passage class, and state 1 is positive recurrent since it is absorbing. Moreover, if we 
partition the matrix P into the 2 x 2 block matrix 


1 0 
[A BI’ 
where 
1 ee 
3 3 3 
Ae eee ele ai 
4 2 4 
we may easily observe that 
7 1 0 
pe= n—-1 Hl 
vy BiA B" 


Since || Bl], = 2, then p(B) < 1, and therefore lim, B” = 0 and S>*) B" = (I— B)1.A 
simple computation leads to 


ee és co 0 0 

: ‘a 22 h2> 9 

lim P"=|1 0 0], R=} >P™=|oo 3 1), 
1 0 0 sees oo 3 2 


in accordance with Facts 3 and 10. 
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2. [BLMO05, Ex. 1.19] The transition matrix 


NIF © 


NIE OO 


Oo Nie 
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is irreducible, and 7 = [, 1,1,...] is an invariant vector. The vector 7 has “infinite” mass, that 
is, the sum of its components is infinite. In fact, the Markov chain is actually null recurrent (see 


[BLMO05, Sec. 1.5]). 


3. [BLMO05, Ex. 1.20] For the transition matrix 


AI © 


Ale OO 


one has 7 P = 7 with 7 = [1, 4, 12, 36, 108,. 


case the Markov chain is transient. 


4. [BLMO05, Ex. 1.21] For the transition matrix 


0 
3 
4 
P= 
0 
= i = 4ysl lid 
one has rP = wm with m = 3[5,3559) 57> 


positive recurrent by Fact 8. 


54.4 Finite Markov Chains 


AIO GO 


CO IW 


..]. The vector 7 has unbounded elements. In this 


So ble 


BIS 


..] and >; zt; = 1. In this case the Markov chain is 


The transition matrix of a finite Markov chain is a finite dimensional stochastic matrix; the reader is 
advised to consult Section 9.4 for properties of finite stochastic matrices. 


Ak x k matrix A is weakly cyclic of index 4 if there exists a permutation matrix TT such that A’ = TTATI? 


Definitions: 
has the block form 
0 
A21 
A’ = 0 
0 


where the zero diagonal blocks are square. 
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1. If P is the transition matrix of a finite Markov chain, there exists an invariant probability vector 7. 
If P is irreducible, the vector 7 is strictly positive and unique, cf. Section 9.4. 

2. For a finite Markov chain no state is null recurrent, and not all states are transient. The states 
belonging to a final class are positive recurrent, and the states belonging to a passage class are 
transient, cf. Section 9.4. 

3. [BP94, Theorem (3.9), Chap. 8] Let P be the transition matrix ofa finite Markov chain. The Markov 
chain is 


(a) Positive recurrent if and only if P is irreducible. 
(b) Regular if and only if P is primitive. 
(c) Periodic if and only of P is irreducible and periodic. 


4. [BP94, Theorem (3.16), Chap. 8] Let P be the transition matrix of a finite irreducible Markov 
chain. Then the Markov chain is periodic if and only if P is a weakly cyclic matrix. 

5. If P is the transition matrix of a regular Markov chain, then there exists lim,_,.) P” = 1a, where 7 
is the probability invariant vector. If the Markov chain is periodic, the sequence { P"},,+0 is bounded, 
but not convergent. 

6. [Mey89] Assume that E = {1,2,...,k} and that P is irreducible. Let 1 < m < k —1 and 
a =({l,...,m},B ={m+1,...,k}. Then 

(a) The matrix I — P[a] is an M-matrix. 
(b) The matrix P’ = Pla] + Pla, B](I — P[B])~!P[B,q@], such that I — P’ is the Schur 
complement of I — P[] in the matrix 


I— Pla] —Pl{a,B] 
—P[p,a] I-—P[p]} 


is a stochastic irreducible matrix. 


(c) If we partition the invariant probability vector 7 as 7 = [7,7], with 7 = (Zj)icq and 
1p = (icp, We have TP! = Ty, 7g = Te Pla, BJ — P[f]). 


Examples: 


1. The matrix 


is the transition matrix of a periodic Markov chain of period 2. In fact, P is an irreducible matrix 
of period 2. 
2. The Markov chain with transition matrix 


ae] 

II 
-——1 
me ule 
Co ule 
[cane 


is regular. In fact, P? > 0,i.e., P is primitive. 


3. Let 
1 1 
ae 
1 1 1 
ee oe ee 0 
= [ibe gy 2 
3 3 3 
1 1 
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be the transition matrix of a Markov chain. The matrix P is irreducible and aperiodic, therefore 
the Markov chain is regular. The vector 7 = a [4, 4, 3, 2] is the stationary distribution. According 
to Fact 5, one has 


[4 4 3 2). 


i= 
3 
ar) 
3 
| 
Se Be eS 


54.5 Censoring 


Definitions: 


Partition the state space E into two disjoint subsets, a and , and denote by {f, t), f,.. .} the time when 
the Markov chain visits the set a: 


ft = min{n > 0: X, € a}, tha = min{n > t +1: Xn € a}, 


for k > 0. The censored process restricted to the subset @ is the sequence {Y,}n+0, where Y,, = X;,, of 
successive states visited by the Markov chain in a. 


Facts: 


1. [BLMO05, Sec. 1.6] The censored process {Y,,} is a Markov chain. 
2. [BLMO05, Sec. 1.6] Arrange the states so that the transition matrix can be partitioned as 


pa| Plal Plas] 
P(p,a] P(g] | 


Then the transition matrix of {Y,,} is 


+00 
P’ = Pla] + 5 (Pla, B)P(B]" P[p,a]) 


n=0 


provided that Se [a, 8] P[B]" P[B,a]) is convergent. If the series S’ = ye P[B]" is con- 
vergent, then we may rewrite P’ as P’ = Pla] + P[a, B]S’P[B, a]. 

3. [BLM05, Theorem 1.23] Assume that the Markov chain is irreducible and positive recurrent. 
Partition the stationary distribution m as 7 = [7,78], with my = (Tj )ieq and Tg = (Ti )icp. 
Then one has wy P’ = 7y, 7p = Toa P[a, B]S’, where P’ and S’ are defined in Fact 2. 

4, [BLM05, Secs. 1.6, 3.5, 4.5] In the case of finite Markov chains, censoring is equivalent to Schur 
complementation (see Fact 6 of Section 54.4). Censoring is at the basis of several numerical 
methods for computing the invariant probability vector 7; indeed, from Fact 3, if the invariant 
probability vector 7g associated with the censored process is available, the vector 7 can be easily 
computed from 74. A smart choice of the sets w and £ can lead to an efficient computation of the 
stationary distribution. As an example of this fact, Ramaswami’s recursive formula for computing 
the vector 7 associated with an M/G/1-type Markov chain is based on successive censorings, where 
the set A is finite, and B = N-a. 
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Numerical Methods 


Definitions: 
Let A beak x k matrix. A splitting A = M — N, where det M # 0, is a regular splitting if M~! > 0 and 


N>=0 


Facts: 
1. 


. A regular splitting is said to be semiconvergent if the matrix M~' N is semiconvergent. 


The main computational problem in Markov chains is the computation of the invariant probability 
vector 77. If the space state E is finite and the Markov chain is irreducible, classical techniques for 
solving linear systems can be adapted and specialized to this purpose. We refer the reader to the book 
[Ste94] for a comprehensive treatment on these numerical methods; in facts below we analyze the 
methods based on LU factorization and on regular splittings of Mo-matrices. If the space state E is 
infinite, general numerical methods for computing 7 can be hardly designed. Usually, when the state 
space is infinite, the matrix P has some structures which are specific of the real problem modeled 
by the Markov chain, and numerical methods for computing 7 which exploit the structure of P can 
be designed. Quite common structures arising in queueing problems are the (block) tridiagonal 
structure, the (block) Hessenberg structure, and the (block) Toeplitz structure (see, for instance, 
Example 4 Section 54.1). We refer the reader to the book [BLM05] for a treatment on the numerical 
solution of Markov chains, where the matrix P is infinite and structured. 

[BP94, Cor. (4.17), Chap. 6], [Ste94, Sec. 2.3] If P isa k x k irreducible stochastic matrix, then the 
matrix I — P has a unique LU factorization I — P = LU, where L is a lower triangular matrix 
with unit diagonal entries and U is upper triangular. Moreover, L and U are My-matrices, U is 
singular, U1, = 0, and (U),, = 0. 


. [Ste94, Sec. 2.5] The computation of the LU factorization of an Mo-matrix by means of Gaussian 


elimination involves additions of nonnegative numbers in the computation of the off-diagonal 
entries of L and U, and subtractions of nonnegative numbers in the computation of the diagonal 
entries of U. Therefore, numerical cancellation cannot occur in the computation of the off-diagonal 
entries of L and U. In order to avoid possible cancellation errors in computing the diagonal entries of 
U, Grassmann, Taksar, and Heyman have introduced in [GTH85] asimple trick, which fully exploits 
the property that U is an Mo-matrix such that Ul, = 0. At the general step of the elimination 
procedure, the diagonal entries are not updated by means of the classical elimination formulas, but 
are computed as minus the sum of the off-diagonal entries. Details are given in Algorithm 1. 

If I — P = LU is the LU factorization of I — P, the invariant probability vector 7 is computed by 
solving the two triangular systems yL = x and xU = 0, where x and y are row vectors, and then 
by normalizing the solution y. From Fact 2, the nontrivial solution of the latter system is x = aep ; 
for any a # 0. Therefore, if we choose a = 1, the vector 7 can be simply computed by solving the 
system yL = e} by means of back substitution, and then by setting 7 = (1/y1)y, as described in 
Algorithm 2. 


. [BP94, Cor. (4.17), Chap. 6] If P isak x k irreducible stochastic matrix, the matrix I — P? has 


a unique LU factorization I — P’ = LU, where L is a lower triangular matrix with unit diagonal 
entries, and U is upper triangular. Moreover, both L and U are Mo-matrices, U is singular, and 
(U)x~ = 0. Since L is nonsingular, the invariant probability vector can be computed by calculat- 
ing a nonnegative solution of the system Uy = 0 by means of back substitution, and by setting 
m= (1/llylli)y" ; 

If P isak x k irreducible stochastic matrix, and if P is the (k — 1) x (k — 1) matrix obtained by 
removing the ith row and the ith column of P, wherei € {1,...,k}, then the matrices I — PT and 
I—Phavea unique LU factorization, where both the factors L and U are M-matrices. Therefore, 
ifi = k and if the matrix I — P? is partitioned as 


I—P™ a 
a Tee 
ms | bt ‘| 
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10. 


11. 


12. 


13: 


14. 


one finds that the vector 7# = [711,...,4-1]7 solves the nonsingular system (I — Pl)# = —ma. 
Therefore, the vector 7 can be computed by solving the system (I — P")y = —a, say by means of 
LU factorization, and then by setting w = [y7,1]/(1 + lly|l1). 


. [Ste94, Sec. 3.6] Let P be a finite irreducible stochastic matrix, and let I — P'’ = M— Nbea 


regular splitting of I — P’. Then the matrix H = M~!N has spectral radius 1 and 1 is a simple 
eigenvalue of H. 


. [Ste94, Theorem 3.3], [BP94], [Var00] Fact 7 above is not sufficient to guarantee that any regu- 


lar splitting of I — P', where P is a finite stochastic irreducible matrix, is semiconvergent (see 
the Example below). In order to be semiconvergent, the matrix H must not have eigenvalues of 
modulus 1 different from 1, and the Jordan blocks associated with 1 must have dimension 1. 


. An iterative method for computing 7 based on a regular splitting I - P’ = M — N consists in 


generating the sequence x,4; = M~'Nx,, for n > 0, starting from an initial column vector xo. 
If the regular splitting is semiconvergent, by choosing xp > 0 such that ||xo||1 = 1, the sequence 
{xn}n converges to 77. The convergence is linear and the asymptotic rate of convergence is the 
second largest modulus eigenvalue @ of M~'N. 

[Ste94, Theorem 3.6] If P is a finite stochastic matrix, the methods of Gauss—Seidel, Jacobi, and 
SOR, for 0 < w < 1, applied to I — P’, are based on regular splittings. 

If E is finite, the invariant probability vector is, up to a multiplicative constant, the nonnegative 
left eigenvector associated with the dominating eigenvalue of P. The power method applied to the 
matrix P’ is the iterative method defined by the trivial regular splitting I - P™ = M — N, where 
M =I, N = P’ | If P is primitive, the power method is convergent. 

[Ste94, Theorems 3.6, 3.7] If P isak x k irreducible stochastic matrix, and if P is the (k—1) x (k—1) 
matrix obtained by removing the ith row and the ith column of P, wherei € {1,...,k}, then any 
regular splitting of the matrix I — P™ = M— N is convergent. In particular, the methods of Gauss— 
Seidel, Jacobi, and SOR, for 0 < w < 1, applied to I — PT, being based on regular splittings, are 
convergent. 

[Ste94, Theorem 3.17] Let P be ak x k irreducible stochastic matrix and let € > 0. Then the 
splitting I — P!’ = M. — N., where Mc = D—L + eI and N, = U+€el,isa semiconvergent 
regular splitting for every € > 0. 

[Ste94, Theorem 3.18] Let P beak x k irreducible stochastic matrix, let I — P' = M—N be 
a regular splitting of I — P?, and let H = M7!N. Then the matrix H, = (1—a@)I + aH is 
semiconvergent for any0 <a <1. 


Algorithms: 


1. 


The following algorithm computes the LU factorization of I — P by using the Grassmann, Taksar, 
Heyman (GTH) trick of Fact 3: 


Computation of the LU factorization of I — P with GTH trick 
Input: the k x k irreducible stochastic matrix P. 
Output: the matrix A = [a;,;] such that a;,; = 1;,; fori > j, 
and aj,; = uj,; fori < j, where L = [];,;], U = [u;,j] are the factors 
of the LU factorization of I — P. 
Computation: 
l-set A= I — P; 
2-form=1,...,k—1: 
(a) fori = m+1,...,k, set djmn = Gim/Amm3 
(b) fori, j =m+1,...,k, set aj,j = aj; — dimam,j ifi A fs 
(c) fori =m+],...,k seta; = — ue aj. 
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2. The algorithm derived by Fact 4 above is the following: 


Computation of 7 through LU factorization 
Input: the factor L = [J;,;] of the LU factorization of the matrix I — P, 
where P isak x k irreducible stochastic matrix. 
Output: the invariant probability vector 7 = (77;);=1,4- 
Computation: 
l-set am, = 1; 
2-for j =k—1,...,lsetzj =-> 
3—set 7 = (3 mj) 1m. 


k dis 
impo Tilip 


Examples: 


The matrix 


0.5 0 0 0.5 
0.2 0.8 O 0 
0 06 0.4 O 
0 0 0.1 0.9 


is the transition matrix of an irreducible and aperiodic Markov chain. The splitting 1 - P’ = M— N 
associated with the Jacobi method is a regular splitting. One may easily verify that the iteration matrix is 


D7! 


r Oo CO CO 


1 0 0 
0 1 0 
0 0 1 
0 0 0 


where D = diag(1, 3, 2, 5). Therefore, the regular splitting is not semiconvergent since the iteration matrix 


has period 4 (see Fact 8 above). 
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Differential equations and differential-algebraic equations arise in numerous branches of science and 
engineering that include biology, chemistry, medicine, structural mechanics, and electrical engineering. 
This chapter is concerned with linear differential(-algebraic) equations with constant coefficients that can 
be analyzed completely via techniques from linear algebra. We discuss the existence and uniqueness of 
solutions of such equations as well as the stability theory. 


55.1 Linear Differential Equations with Constant 
Coefficients: Basic Concepts 


Definitions: 
A linear differential equation in an unknown function x : RR > C,tb x(t), has the form 
x=ax+ fy 
where a € C and the inhomogeneity f : R + Cisa given function. Here x denotes the derivative of x(t) 
with respect to tf. 
A linear differential equation of order k for an unknown function x : R > C has the form 
apx” + Brae + Agx = ip 


where dy,..., a, € C, a, 4 Oand f : RR > C. Here x denotes the k-th derivative of x(t) with respect 
tot. 

A system of linear differential (-algebraic) equations with constant coefficients has the form 

Ex= Ax+f, 

where E, A € C’”™" are coefficient matrices, x : R — C” is a vector-valued function of unknowns, and 
the inhomogeneity f : R + C” is a given vector-valued function. 

If E =I, and A € C”*", then Ex = Ax + fisa system of ordinary differential equations; otherwise 
it is a system of differential-algebraic equations. 
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A homogeneous system has f(t) = 0; otherwise the system is inhomogeneous. 

A system of linear differential-algebraic equations of order k for an unknown function x : R > C" 
has the form Ayx) + .---+ Aox = f, where Ao,..., Ag € C”*", Ay 4 Oandf:R—> C”. 

A continuously differentiable function x : R + C” is a solution of Ex = Ax + f with a sufficiently 
often differentiable function f if it satisfies the equation pointwise. 

A solution of Ex = Ax + f that also satisfies an initial condition x(to) = xy with tf) € R and xp € C” is 
a solution of the initial value problem. 

If the initial value problem Ex = Ax + f, x(f) = Xo has a solution, then the initial condition is called 
consistent. 


Facts: 


1. [Cam80, p. 33] Let E, A € C”*”. If E is nonsingular, then the system Ex = Ax + f is equivalent 
to the system of ordinary differential equations x = E~' Ax + E~'f. 
2. [Arn92, p. 105] The k-th order system of differential-algebraic equations 


Apy® +--+ + Aoy =¢ 


can be rewritten as a first-order system Ex = Ax + f, where f = [0, ..., 0, gt ]? and 
I 0 I 0 y 
E= , A= ,x= 
I O « 0 I yk?) 
Ay —Ay Aga — AR yen 
Applications: 


1. Consider a mass—spring—damper model as shown in Figure 55.1. This model is described by the 
equation 


mx +dx+kx =0, 


where m is a mass, k is a spring constant, and d is a damping parameter. Since m # 0, we obtain 
the following first-order system of ordinary differential equations 


PI Lire -al bh 


where the velocity is denoted by v. 


FIGURE 55.1 A mass—spring—damper model. 
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2. Consider a one-dimensional heat equation 


2 


0 0 
5p 1 8) = Capi 5), (t,&) e (0, te) x (0,1), 


together with an initial condition T(0,&) = g(&) and Cauchy boundary conditions 


ay T(t,0) + o~-T(t,0) = u(t), 
on 

BiT(t,1) + Be—-T(bl) = v(0). 
on 


Here T(t, &) is the temperature field in a thin beam of length /, c > 0 is the heat conductivity of the 


material, g(&), u(t), and v(t) are given functions, and 2 denotes the derivative in the direction of 


the outward normal. A spatial discretization by a finite difference method with n + 1 equidistant 
grid points leads to the initial value problem x = A, »x + f, x(0) = xo, where 


x(t) =[T(t,h), T(t,2h), ..., T(t,nh)]", xo =[g(h), g(2h), ..., g(nh)]', 


f(t) = [cu(t)/(h?o — hon), 0, ..., 0, ev(t)/(h?B; + hB2) I", 


and 


Ago = > a fs oe eR" 


with h =1/(n + 1), a = (2ha — a) /(hoy — a), and b = (2hB, + Br)/(hBi + By). 
3. A simple pendulum as shown in Figure 55.2 describes the movement of a mass point with mass m 
and Cartesian coordinates (x, y) under the influence of gravity in a distance / around the origin. 


FIGURE 55.2 A simple pendulum. 
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The equations of motion have the form 


mx + 2xr’ = 0, 
my + 2ya + mg =0, 
e+y—P=0, 
where A is a Lagrange multiplier. Transformation of this system into the first-order form by intro- 


ducing new variables v = x and w = y and linearization at the equilibrium x, = 0, ye = —I, 
Ve = 0, we = 0, anda, = mg/(21) yields the homogeneous first-order linear differential-algebraic 


system 
iO! 0 02 0] T* 0 0 10 % 
010 0 01 l¥ 0 0 01 ol 17 
00m 0 o| |¥}|=]-2,. 0 0 0 of JF], 
00 0 m Of lw 0 —2r2, 0 0 2I| |w 
00 0 0 Of |x O° =o (0:0: 0) 1% 


where X =X —X%5,V=V—-VeV=V—-Yy,W=W weandA =A—Ay. 
The motion of the pendulum can also be described by the ordinary differential equation 


g= —w sin(g), 


where ¢ is an angle between the vertical axis and the pendulum and w = /g/I is an angular 
frequency of the motion. By introducing a new variable y = @ and linearization at the equilibrium 
ge = Oand ¥, = 0, we obtain the first-order homogeneous system 


Consider a simple RLC electrical circuit as shown in Figure 55.3. Using Kirchoff’s and Ohm’s laws, 
the circuit can be described by the system Ex = Ax + f with 


L 00 0 0 10 0 i 
001 0 WE WOo 0-0 VL 0 
E= , A= ,x= , f= 
0 0 0 0 SRE O07 4 Ve 0 
0 0 0 0 Oe. Tee 4 VR -v 


Here R, L, and C are the resistance, inductance, and capacitance, respectively; vr, vz, and vc are the 
corresponding voltage drops, i is the current, and v is the voltage source. From the last two equations 


=) = —C 


FIGURE 55.3 A simple RLC circuit. 
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in the system, we find vg = Ri andv,; = v — vc — Ri. Substituting v, in the first equation and 
introducing a new variable wc = i/C, we obtain the system of ordinary differential equations 


VC = 0 1 VC 0 
we| |-1/(LC) —R/L| |we + v/(LC)|" 


This shows the relationship to the mass—spring—damper model as in Application 1. 


55.2 Linear Ordinary Differential Equations 


Facts: 


The following facts can be found in [Gan59a, pp. 116-124, 153-154]. 


1. Let J4 = T~!AT be in Jordan canonical form. Then e“’ = Te/*'T—!. (See Chapter 6 and 
Chapter 11 for more information on the Jordan canonical form and the matrix exponential.) 

2. Every solution of the homogeneous system x = Ax has the form x(t) = e“‘v with v € C”. 

3. The initial value problem x = Ax, x(t) = Xo has the unique solution x(t) = e4—") x9. 

4. The initial value problem x = Ax + f, x(f) = Xp has a unique solution for every initial vector xo 
and every continuous inhomogeneity f. This solution is given by 


t 
x(t) = eA’) x, + eAl—D Er) dr. 
to 


Examples: 
1. Let 
3 3 1 1 —3t? — 3t 
A=] 0 0 Of, Xo = |21, f(t) = 2t 
-1 -1 3 eP+t—1 
For 
1 -1 1 10 
T=]! 0 1 and Tli=]1 1 1], 
= 0 0 1 0 
we have 
2 1 0 
Ja=TIAT=|0 2 0 
00/0 
Then 
1 
ee = 
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Every solution of the homogeneous system x = Ax has the form 


(1+ t)y, + (1+ t)v2 + tv3)e7 — vp 
x(t) = e“v= V2 


(—tv,; — tv. + (1 — t)v3)e”" 
with v = [v1, V2, v3]". The solution of the initial value problem x = Ax, x(0) = Xp has the form 


(3+ 6t)e2? — 2 
x(t) = e“'x) = 2 
(3 — 6t)e?! 


The initial value problem x = Ax + f, x(0) = Xo has the solution 


(3+ 6t)e*+t—2 
x(t) = ?+2 
(3 — 6t)e* +1 


Applications: 


1. Consider the matrix A from the mass—spring—damper example 


The Jordan canonical form of A is given by J4 = T~!AT = diag(A1,A2), where 


SW = 1 ae | 
pS pee 
Ar Ag Az2— Ar |—-Ay 1 


and 
4 —d — Jd? —4km 4 —d+ Jd? —4km 
c= 5) > 
2m 2m 


are the eigenvalues of A. We have 


1 anem! — red! giat — phit 
e“' = Tdiag(e™’,e)T | = ' ' ; . seals 
Ar — Aq | AyAg(e** — e*2*)  Age*2* — Aye” 


The solution of the mass—spring—damper model with the initial conditions x(0) = xp and v(0) = 0 
is given by 


Xo 
A2— Ay 


x(t) = (Are™* _ Ae"), v(t) = ———— (em — ge): 

2. Since the matrix A,» in the semidiscretized heat equation is symmetric, there exists an orthogonal 
matrix U such that UT Aap U = diag(A1,...,An), where A1,...,An € Rare the eigenvalues of Ag,p. 
(See Chapter 7.2 and Chapter 45.) In this case e4*' = Udiag(e*"’,...,e*!)U?. 
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55.3 Linear Differential-Algebraic Equations 


Definitions: 


A Drazin inverse A? of a matrix A € C"*” is defined as the unique solution of the system of matrix 
equations 


AP AAP = AP, AAP = APA, Att AP — ak 


where k is a smallest nonnegative integer such that rank( A*+!) = rank(A*). 
Let E, A € C”*”. A pencil of the form 


AE — A= diag (Lis. + Laps Mims ---> Mags Tes No) 


is called pencil in Kronecker canonical form if the block entries have the following properties: every entry 
Ln; =ALn, — Rn, isa bidiagonal block of size n; x (nj + 1),nj € N, where 


everyentryM in, = ML “Ri, isa bidiagonal block of size (mj+1)xmj,m; € N;theentry % = Alp—Ax 
is a block of size k x k, k € N, where A; is in Jordan canonical form; the entry VW, = AN, — I, is a block 
of sizes x s,s € N, where N, = diag(N,,,..., Ns, ); and 


0 1 
N,, = 
J 
1 
0 
is a nilpotent Jordan block with index of nilpotency sj. 

The numbers 1,..., 1p are called the right Kronecker indices of the pencil AE — A. 
The numbers ,..., mq are called the left Kronecker indices of the pencil AE — A. 


The number v = max)<j<, s; is called the index of the pencil AE — A. 

A matrix pencil AE — A with E, A € C””" is called regular, if m = n and det(AE — A) 4 0 for some 
A € C. Otherwise, the pencil is called singular. 

Let E, Ae C”". Subspaces WH C C” and W, C C” are called left and right reducing subspaces of the 
pencilAE — Aif W = EW, + AW, and dim(W)) = dim(W,) — p, where p is the number of Ly, blocks 
in the Kronecker canonical form. 

Let AE — A bea regular pencil. Subspaces W, W, C C” are called left and right deflating subspaces 
of LE — AifW = EW, + AW, and dim(W) = dim(W,). 

Let Wi, W2 Cc C” be subspaces such that W, MN W) = {0} and W, + Wy = C”. A matrix P € C”” is 
called a projection onto W, along W, if P? = P, range(P) = W,, and ker(P) = W). 

Let AE — A bea regular pencil. If T;, T. € C"*" are nonsingular matrices such that T, (AE — AJT, is 
in Kronecker canonical form, then 
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are the spectral projections onto the left and right deflating subspaces of AE — A corresponding to the 
finite eigenvalues along the left and right deflating subspaces corresponding to the eigenvalue at infinity. 


Facts: 


1. 
2. 


[Cam80, p. 8] If A € C’*” is nonsingular, then AP? = AW}. 


[Cam80, p. 8] Let 
» A, 0 
Ja=T AT= 
0 Apo 


be in Jordan canonical form, where A, contains all the Jordan blocks associated with the nonzero 
eigenvalues, and Ag contains all the Jordan blocks associated with the eigenvalue 0. Then 


-1 
AP =T a 7, 
0 Oo 


s an 3 5 a or every matrix pencil = wit ’ E there exist nonsingular 
[Gan59b, pp. 29-37] F y matrix pencil AE — A with E, A € C’”*" there exi ingul 


matrices T; € C”*” and T, € C"*" such that TOE — AJT, is in Kronecker canonical form. 
The Kronecker canonical form is unique up to permutation of the diagonal blocks, i.e., the kind, 
size, and number of the blocks are characteristic for the pencil AE — A. (For more information on 
matrix pencils, see Section 43.1.) 


. [Gan59b, p. 47] If f(t) = [fi(t),..., tn; (t)]" is ann j-times continuously differentiable vector- 


valued function and g(t) is an arbiter (nj; + 1)-times continuously differentiable function, then 
the system Dy,X = RajX + f has a continuously differentiable solution of the form 


g(t) 
eV) - fl 


xm — [g(t — Oo FP) 
i=1 


A consistent initial condition has to satisfy this defining equation at fo. 


. [Gan59b, p. 47] A es of differential-algebraic equations Linx = Rin X + f with a vector- 


valued function f(t) = [fi(t),. rg ee a has a unique solution if and only if f is mj-times 
mj+1 

continuously differentiable and x an "(t) = 0. If this holds, then the solution is given by 
i=l 


[2 J >> re 


x(t) = : | = 
Xm (E) AFgele) 


A consistent initial condition has to satisfy this defining equation at fo. 


. [Gan59b, p. 48] A system N,,x = x + fhas a unique continuously differentiable solution x if f is 


s ;-times continuously differentiable. This solution is given by 


sj-l 


=-EN 0. 


A consistent initial condition has to satisfy this defining equation at fo. 
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7. [Cam80, pp. 37-39] If the pencil XE — A is regular of index v, then for every v-times differentiable 
inhomogeneity f there exists a solution of the differential-algebraic system Ex = Ax + f. Every 
solution of this system has the form 

A A t A a ve 
x(t) = Phew p rey | eb At) BPR) dt — (I — BPE) S (BA?) APE (t), 
to j=0 
where v € C”, EF = (AgE — A)"!E, A= (ApE — A)~A, and f = (AgE — A)~!f for some Ay € C 
such that Ag E — A is nonsingular. 

8. [Cam80, pp. 37-39] Ifthe pencil AE — A is regular of index v and if f is v-times differentiable, then 
the initial value problem Ex = Ax + f, x(f) = xo possesses a solution if and only if there exists 
v € C" that satisfies 

v-l 
Xo = EPEv—(1— BPE) S (BA?) APE? (a) 
j=0 
If such a v exists, then the solution is unique. 

9. [KM06, p. 21] The existence of a unique solution of Ex = Ax + f, x(fo) = xo does not imply that 

the pencil AE — A is regular. 
10. [Cam80, pp. 41-44] If the pencil AE — A is singular, then the initial value problem Ex = Ax+ f, 
x(to) = Xo may have no solutions or the solution, if it exists, may not be unique. 
11. [Sty02, pp. 23-26] Let the pencil AE — A be regular of index v. If 
- AI—Ap 0 
T, AE-A)T, = 
0 ANT 
is in Kronecker canonical form, then the solution of the initial value problem Ex = Ax + f, 
X(t) = Xo can be represented as 
t v-l 
x(t) = F(t — t) Exo +f Ft —c)flr)de+ $0 F_j-f (0), 
t j=0 
where 
eM UD | 0 0 3 
F(t) =T, pi ST ea 3 
Examples: 
1. The system 
10 5c) 0) 4b ae, 1 
x= x ’ >.< = 
0 0 0 0 g(t) 0 
has no solution if g(t) 4 0. For g(t) = 0, this system has the solution x(t) = [e’, (t) ]", where 
(t) is a differentiable function such that #(0) = 0. 
2. The system 

1 0 0 0 — sin(t) ; 

0 1l1)x=1]1 O| x+ |—cos(t) ; o=[ 

0 0 0 1 0 


has a unique solution x(t) = [ cos(t), 0 |", but the pencil AE — A is singular. 
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Applications: 


1. The pencil in the linearized pendulum example has the Kronecker canonical form 


10/00 0 —iJ/g/T 0 0 0 0 
0 1] 0 0 0 0 i/g/T | 0 0 0 
diag(A,N3)=A}0 0 | 0 1 Of - 0 0 1 0 Of. 
OE) AG 20> a 0 0 0 1 0 
00/0 0 0 0 0 001 


This pencil is regular of index 3. Since the linearized pendulum system is homogeneous, it has 
a unique solution for every consistent initial condition. 
2. The pencil of the circuit equation has the Kronecker canonical form 


ae a Bee Be | 8 

; On |) Oe 0 ee ORME 
diag(Jr,N2) = = , 

001/00 0 O 1 0 

Oat |. OG Oe! |) ok 

with 

: R R2 1 : RL R2 1 
Po aE 402 - EC’ cme |) AL? BC 


This pencil is regular of index 1. Hence, there exists a unique continuous solution for every con- 
tinuous voltage source v(t) and for every consistent initial condition. 


55.4 Stability of Linear Ordinary Differential Equations 


The notion of stability is used to study the behavior of dynamical systems under initial perturbations 
around equilibrium points. In this section, we consider the stability of linear homogeneous ordinary 
differential equations with constant coefficients only. For extensions of this concept to general nonlinear 
systems, see, e.g., [Ces63] and [Hah67]. 


Definitions: 


The equilibrium x,(t) = 0 of the system x = Ax is called stable in the sense of Lyapunov, or simply 
stable, if for every ¢ > 0 there exists a5 = 5(€) > 0 such that any solution x of k = Ax, x(f) = Xo with 
xoll2 < 6 satisfies ||x(f)||2 < ¢ for allt > t. 

The equilibrium x,(t) = 0 of the system x = Ax is called asymptotically stable if it is stable and 
iim x(t) = 0 for any solution x of x = Ax. 


The equilibrium x,(t) = 0 of the system x = Ax is called unstable if it is not stable. 
The equilibrium x,(t) = 0 of the system x = Ax is called exponentially stable if there exist w > 0 and 
B > O such that the solution x of x = Ax, x(t) = Xo satisfies ||x(t)||2 < a e~F"—®) ||xo ||, for all t > fp. 


Facts: 


1. [Gan59a, pp. 125-129] The equilibrium x,(t) = 0 of the system x = Ax is stable if and only if all 
the eigenvalues of A have nonpositive real part and those with zero real part have the same algebraic 
and geometric multiplicities. If at least one of these conditions is violated, then the equilibrium 
xX,(t) = 0 of k = Axis unstable. 

2. [Gan59a, pp. 125-129] The equilibrium x, (t) = 0 of the system x = Ax is asymptotically stable if 
and only if all the eigenvalues of A have negative real part. 
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3. [Ces63, p. 22] Let pa(A) = det(AI — A) = A" +.a,A""!+---+., be the characteristic polynomial 
of A € R"”". Ifthe equilibrium x, (t) = 0 of the system x = Axis asymptotically stable, then a; > 0 
for j =1,...,n. 

4, [Gan59b, pp. 185-189] The equilibrium x,(t) = 0 of the system x = Ax is asymptotically stable 
if and only if the Lyapunov equation A* X + XA = —Q has a unique Hermitian, positive definite 
solution X for every Hermitian, positive definite matrix Q. 

5. [God97] Let H be a Hermitian, positive definite solution of the Lyapunov equation 


A,H+HA=-I 


and let x bea solution of the initial value problem x = Ax, x(0) = xo. Then in terms of the original 
data, 


IIx(t)l2 < VK (A) eMAle/«A) x0 II, 


where «(A) = 2||All2|| H]l2. 

6. [Hah67, pp. 113-117] The equilibrium x,(t) = 0 of the system x = Ax is exponentially stable if 
and only if it is asymptotically stable. 

7. [Hah67, p. 16] If the equilibrium x,(t) = 0 of the homogeneous system x = Ax is asymptot- 
ically stable, then all the solutions of the inhomogeneous system x = Ax + f with a bounded 
inhomogeneity f are bounded. 


Examples: 
1. Consider the linear system x = Ax with 


For the initial condition x(0) = [1, 0]’, this system has the solution x(t) = [e~‘, 0 |" that is 
bounded for all t > 0. However, this does not mean that the equilibrium x,(t) = 0 is stable. For 
linear systems with constant coefficients, stability means that the solution x(t) remains bounded for 
all time and for all initial conditions, but not just for some specific initial condition. If we can find 
at least one initial condition that causes one of the states to approach infinity with time, then the 
equilibrium is unstable. For the above system, we can choose, for example, x(0) = [ 1, 1]". In this 
case x(t) = [e~', e”" |? is unbounded, which proves that the equilibrium x, (t) = 0 is unstable. 
2. Consider the linear system x = Ax with 


The eigenvalues of A are —0.1 + i, and, hence, the equilibrium x, (t) = 0 is asymptotically stable. 
Indeed, the solution of this system is given byx(t) = e4'x(0), which can be written in the real formas 


x;(t) = e°'(x1(0) cos(t) — x2(0) sin(t)), 
xo(t) = e7° "(x1 (0) sin(t) + x2(0) cos(t)). 


(See also Chapter 56.) Thus, for all initial conditions x,(0) and x2(0), the solution tends to zero as 
t — oo. The phase portrait for x;(0) = 1 and x2(0) = 0 is presented in Figure 55.4. 
3. Consider the linear system x = Ax with 
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FIGURE 55.4 Asymptotic stability. 


The matrix A has the eigenvalues +i. The solution of this system x(t) = e“*x(0) can be written in 
the real form as 


x, (t) = x, (0) cos(t) — x2(0) sin(t), 
xX2(t) = x,(0) sin(t) + x2(0) cos(t). 
It remains bounded for all initial values x, (0) and x2(0), and, hence, the equilibrium x,(t) = 0 is 


stable. The phase portrait for x;(0) = 1 and x2(0) = 0 is given in Figure 55.5. 
4. Consider the linear system x = Ax with 


0.1 —-1 
A= ‘ 
1 0.1 


The eigenvalues of A are 0.1 +7. The solution of this system in the real form is given by 


x(t) = e° (x, (0) cos(t) — x2(0) sin(¢)), 
x2(t) = e°!(x1(0) sin(t) + x2(0) cos(t)). 
It is unbounded for all nontrivial initial conditions. Thus, the equilibrium x,(t) = 0 is unstable. 


The phase portrait of the solution with x|(0) = 1 and x)(0) = 0 is shown in Figure 55.6. 
5. Consider the linear system x = Ax with 


a 
=| 
c 


The characteristic polynomial of the matrix A is given by p,(A) = A? — (a+. d)A + (ad — bc) and 
the eigenvalues of A have the form 


€ R2*2. 


at+d | J(a +d)? — 4(ad — bc) ¢ at+d JSa+d) — 4(ad — bc) 
T 5) Qa . 


A= 
2 2 2 2 
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FIGURE 55.5 Stability. 


FIGURE 55.6 Instability. 
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We have the following cases: 


a+d <0, ad—bc>0 Re(A,) < 0, Re(A2) < 0, Asymptotically stable 
a+d <0, ad—bc=0 A, =0,A2 < 0 Stable 
a+d<0, ad—bce <0 dy >0,A2 < 0 Unstable 
a+d=0, ad—bc>0 dy = ia, Az = —ia, a - real Stable 
a+d=0, ad—bc=0 A, =0,A2 = 0 Unstable 
7+hV4+e4+d*40 

a=0,b3%,¢=0, 2=0 A, =0,A2 = 0 Stable 
a+d=0, ad—bc <0 dy >0,A2 < 0 Unstable 
a+d>0, ad—bc <0 A, > 0,A2 < 0 Unstable 
a+d>0, ad—bc>0 Re(A,) > 0, Re(A2) > 0 Unstable 


Applications: 


1. Consider the semidiscretized heat equation; see Application 2 in Section 55.1. Leta, = 6, = 1 
and a = B, = 0. Then a = b = 2 and the matrix A), has the eigenvalues 
4c ju 


4j(Az2) = in? 
Maa) So Fat) 


In this case, the equilibrium x,(t) = 0 of the system x = A»,x is asymptotically stable. However, 
for a, = 6; = 0 and a = fo = 1, we have a = b = 1. Then the matrix A,,, has a simple zero 
eigenvalue and, hence, the equilibrium x, (t) = 0 of k = Aj)x is only stable. 

2. Consider the mass—spring—damper model with m > 0, d > 0, and k > 0. The coefficient matrix 
of this model has eigenvalues 


—d-VP—tkm d+ VP — 4m 


Xx =S — 
: 2m 2m 


For d = 0, the equilibrium x, (t) = 0 is unstable if k = 0, and it is stable if k > 0. Ford > 0, the 
equilibrium x,(t) = 0 is stable if k = 0, and it is asymptotically stable if k > 0. 


55.5 Stability of Linear Differential-Algebraic Equations 


Definitions: 


The equilibrium x,(t) = 0 of the system Ex = Ax is called stable in the sense of Lyapunov, or simply 
stable, if for every ¢ > 0 there exists ad = 6(¢) > 0 such that any solution x of Ex = Ax, x(to) = P,X 
with || P,xXo||2 < 6 satisfies ||x(t)||.2 < ¢ for allt > t. 

The equilibrium x,(t) = 0 of the system Ex = Ax is called asymptotically stable if it is stable and 
iim x(t) = 0 for every solution x of Ex = Ax. 


7 The equilibrium x, (t) = 0 of the system Ex = Ax is called unstable if it is not stable. 

The equilibrium x,(t) = 0 of the system Ex = Axis called exponentially stable if there exist constants 
a > Oand B > Osuch that the solution x of Ex = Ax, x(fo) = P,Xo satisfies ||x(t) ||, < a e~F"— || P, xo 
for all t > to. 


Facts: 


1. [Dai89, pp. 68-69] If the pencil AE — A is regular, then the equilibrium x,(t) = 0 of the system 
Ex = Axis stable if and only if all finite eigenvalues of the pencil AE — A have nonpositive real 
part and those with zero real part have the same algebraic and geometric multiplicities. 
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2. [Dai89, pp. 68-69] If the pencil AE — A is regular, then the equilibrium x, (t) = 0 of the system 
Ex = Ax is asymptotically stable if and only if all finite eigenvalues of AE — A have negative real 
part. 

3. [Sty02, p. 48] Let Q be a Hermitian matrix such that v‘;Qv > 0 for all nonzero vectors 
v € range(P,). The equilibrium x,(t) = 0 of Ex = Ax is asymptotically stable if the general- 
ized Lyapunov equation E*X A+ A* XE = —Qhasa Hermitian, positive semidefinite solution X. 

4. [Sty02, pp. 49-52] The equilibrium x, (t) = 0 of the system Ex = Ax is asymptotically stable and 
the pencil AE — A is of index at most one if and only if the generalized Lyapunov equation E* X A+ 
A*XE = —E*QE, with Hermitian, positive definite Q has a Hermitian, positive semidefinite 
solution X. 

5. [TMK95] The equilibrium x, (t) = 0 of the system Ex = Axis asymptotically stable and the pencil 
AE — Ais of index at most one if and only if the generalized Lyapunov equation 


A*X+Y*A=—-Q, Y*E = E*X, 


with Hermitian, positive definite Q has a solution (X,Y) such that E*X is Hermitian, positive 
semidefinite. 

6. [Sty02, pp. 52-54] The equilibrium x,(t) = 0 of the system Ex = Ax is asymptotically stable if 
and only if the projected generalized Lyapunov equation 


E*XA+A*XE=—P*QP,, X= P*XP, 


has a unique Hermitian, positive semidefinite solution X for every Hermitian, positive definite 
matrix Q. 

7. [Sty02] The equilibrium x,(t) = 0 of the system Ex = Ax is asymptotically stable if and only if 
the projected generalized Lyapunov equation 


EYA*+AYE*=—P,QP;*, Y=P.YP* 


has a unique Hermitian, positive semidefinite solution Y for every Hermitian, positive definite 
matrix Q. 

8. [Sty02, pp. 28-31] Let H be asymmetric, positive semidefinite solution of the projected generalized 
Lyapunov equation E*HA+ A*HE = —P*P,, H = PHP, and let x bea solution of the initial 
value problem Ex = Ax, x(0) = P,xo. Then in terms of the original data, 


IIx(t)|l2 < V/«(E, ADE |lol|(EP, + AC — PH fp eA AIE I) | D xo lla, 


where k(E, A) = 2||E|l2|| All2|| All2- 
9. From the previous fact it follows that the equilibrium x,(t) = 0 of the system Ex = Ax is 
exponentially stable if and only if it is asymptotically stable. 


Applications: 


1. The finite eigenvalues of the pencil AE — A in the RLC electrical circuit example are given by 


Hence, the equilibrium x, (t) = 0 of the system Ex = Ax is asymptotically stable. 

2. The pencil AE — A in the linearized pendulum example has the finite eigenvalues 4; = —i./g/T 
and Ay = i./g/T. In this case the equilibrium x, (t) = 0 of the system Ex = Ax is stable but not 
asymptotically stable. 
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Examples: 
1. The generalized Lyapunov equation E*X A + A* XE = —Q with 


1 0 
E= , 
0 0 
has no solution, although the finite eigenvalue of AE — A is negative and AE — A has index one. 
2. The generalized Lyapunov equation E*X A+ A*XE = —E*QE with 


1 0 0 =) 0 1 0 0 
E=]0 0 1|, A= ol, Q=|0 2 0 
0 0 O 1 0 0 2 
has no Hermitian, positive semidefinite solution, although the finite eigenvalue of AE — A is 


negative. 
3. The generalized Lyapunov equation A*X + Y*A = —Q, Y*E = E*X with 


1 0 0 —-1 0 0 1 0 0 
E=1{0 0 1], A= 0 1 O;,; Q=]0 1 0 
0 0 0 0 0 1 0 0 1 


has no solution, although the finite eigenvalue of AE — A is negative. 
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Linear algebra plays a key role in the theory of dynamical systems, and concepts from dynamical systems 
allow the study, characterization, and generalization of many objects in linear algebra, such as similarity of 
matrices, eigenvalues, and (generalized) eigenspaces. The most basic form of this interplay can be seen as 
a matrix A gives rise to a continuous time dynamical system via the linear ordinary differential equation 
x = Ax, or a discrete time dynamical system via iteration x,;, = Ax,. The properties of the solutions 
are intimately related to the properties of the matrix A. Matrices also define nonlinear systems on smooth 
manifolds, such as the sphere S4-! in R@, the Grassmann manifolds, or on classical (matrix) Lie groups. 
Again, the behavior of such systems is closely related to matrices and their properties. And the behavior 
of nonlinear systems, e.g., of differential equations y = f(y) in R@ with a fixed point yo € R’, can be 
described locally around yo via the linear differential equation x = Dy f (yo)x. 

Since A. M. Lyapunov’s thesis in 1892, it has been an intriguing problem how to construct an appro- 
priate linear algebra for time varying systems. Note that, e.g., for stability of the solutions of x = A(t)x, 
it is not sufficient that for all tf € R the matrices A(t) have only eigenvalues with negative real part 
(see [Hah67], Chapter 62). Of course, Floquet theory (see [Flo83]) gives an elegant solution for the peri- 
odic case, but it is not immediately clear how to build a linear algebra around Lyapunov’s “order numbers” 
(now called Lyapunov exponents). The multiplicative ergodic theorem of Oseledets [Ose68] resolves the 
issue for measurable linear systems with stationary time dependencies, and the Morse spectrum together 
with Selgrade’s theorem [Sel75] clarifies the situation for continuous linear systems with chain transitive 
time dependencies. 

This chapter provides a first introduction to the interplay between linear algebra and analysis/topology 
in continuous time. Section 56.1 recalls facts about d-dimensional linear differential equations x = Ax, 
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emphasizing eigenvalues and (generalized) eigenspaces. Section 56.2 studies solutions in Euclidian space 
R¢ from the point of view of topological equivalence and conjugacy with related characterizations of the 
matrix A. Section 56.3 presents, in a fairly general set-up, the concepts of chain recurrence and Morse 
decompositions for dynamical systems. These ideas are then applied in section 56.4 to nonlinear systems on 
Grassmannian and flag manifolds induced by a single matrix A, with emphasis on characterizations of the 
matrix A from this point of view. Section 56.5 introduces linear skew product flows as a way to model time 
varying linear systems xX = A(t)x with, e.g., periodic, measurable ergodic, and continuous chain transitive 
time dependencies. The following sections 56.6, 56.7, and 56.8 develop generalizations of (real parts of) 
eigenvalues and eigenspaces as a starting point for a linear algebra for classes of time varying linear systems, 
namely periodic, random, and robust systems. (For the corresponding generalization of the imaginary 
parts of eigenvalues see, e.g., [Arn98] for the measurable ergodic case and [CFJ06] for the continuous, 
chain transitive case.) Section 56.9 introduces some basic ideas to study genuinely nonlinear systems 
via linearization, emphasizing invariant manifolds and Grobman—Hartman-type results that compare 
nonlinear behavior locally to the behavior of associated linear systems. 


Notation: 
In this chapter, the set of d x d real matrices is denoted by g/(d, R) rather than R?*“. 


56.1 Linear Differential Equations 


Linear differential equations can be solved explicitly if one knows the eigenvalues and a basis of eigenvectors 
(and generalized eigenvectors, if necessary). The key idea is that of the Jordan form of a matrix. The real 
parts of the eigenvectors determine the exponential behavior of the solutions, described by the Lyapunov 
exponents and the corresponding Lyapunov subspaces. 

For information on matrix functions, including the matrix exponential, see Chapter 11. For information 
on the Jordan canonical form see Chapter 6. Systems of first order linear differential equations are also 
discussed in Chapter 55. 


Definitions: 
Fora matrix A € gl(d, R), the exponential e“ ¢ GL(d, R) is defined bye“ = 1+ 7, 4A" € GL(d,R), 
where I € gl(d,R) is the identity matrix. 

A linear differential equation (with constant coefficients) is given by a matrix A € gl(d,R) via 
x(t) = Ax(t), where x denotes differentiation with respect to t. Any function x : R —> IR? such that 
x(t) = Ax(t) for all t € R is called a solution of k = Ax. 

The initial value problem for a linear differential equation x = Ax consists in finding, for a given initial 
value x) € R4, a solution x(-, x9) that satisfies x(0, x9) = Xo. 

The distinct (complex) eigenvalues of A € gl(d,R) will be denoted j1,..., 4,. (For definitions and 
more information about eigenvalues, eigenvectors, and eigenspaces, see Section 4.3. For information 
about generalized eigenspaces, see Chapter 6.) The real version of the generalized eigenspace is denoted 
by E(A, ue) C Ré or simply E; fork =1,...,r <d. 

The real Jordan form of a matrix A € gl(d,R) is denoted by J ve Note that for any matrix A there is a 
matrix T € GL(d,R) such that A= T'J®T. 

Let x(-, Xo) be a solution of the linear differential equation x = Ax. Its Lyapunov exponent for x) 4 0 
is defined as A(xo) = lim sup,_, ,, ¢ log ||x(t, Xo) ||, where log denotes the natural logarithm and || - || is any 
norm in R¢, 

Let py = Ag + ivy, k = 1,...,17, be the distinct eigenvalues of A € gl(d,R). We order the distinct 
real parts of the eigenvalues as 4; < ... < 43,1 <1 <r < d, and define the Lyapunov space of 1; as 
L(A;) = ® Ex, where the direct sum is taken over all generalized real eigenspaces associated to eigenvalues 
with real part equal to 4 ;. Note that Bi-1 Lj) =R¥*. 


Dynamical Systems and Linear Algebra 56-3 


Pais stable, center, and unstable subspaces associated with the matrix A € gl(d,R) are defined as 
= P{L(A;), A; < 0}, L° = Q{L(A;), Aj = 0}, and Lt = @{L(Aj), Aj > O}, respectively. 
om Zero solution x(t,0) = 0 is Galea eiponentially stable if there exists a neighborhood U(0) and 
positive constants a,b > 0 such that x(t, x9) < al|xolle~"' for all t € R and x» € U(0). 


Facts: 
Literature: [Ama90], [HSD04]. 


1. For each A € gl(d,R) the solutions of x = Ax form a d-dimensional vector space sol(A) C 
C™(R, R?) over R, where C~(R, R*) = { f:R— R¢, f is infinitely often differentiable}. Note 
that the solutions of x = Ax are even real analytic. 

2. For each initial value problem given by A € gl(d,R) and xp € 4, the solution x(-,x9) is unique 
and given by x(t, Xp) = e4"x. 

3. Let vj,...,Vy € R?@ bea basis of R?. Then the functions x(-,v,),...,x(-, vg) form a basis of the 
solution space sol(A). The matrix function X(-) := [x(-,v,),...,x(-, Vg)] is called a fundamental 
matrix of x = Ax, and it satisfies X(t) = AX(t). 

4. Let A € gl(d,R) with distinct eigenvalues jz,..., 4, € C and corresponding multiplicities n, = 
a(jx), k = 1,...,1r. If E, are the corresponding generalized real eigenspaces, then dim E, = nx 
and Bie E;, = R4,ie., every matrix has a set of generalized real eigenvectors that form a basis of 
R¢ 


5. IfA= Tope T, then e4* = TO elatT, ie., for the computation of exponentials of matrices it is 
sufficient to know the exponentials of Jordan form matrices. 

6. Let vj,...,Vvq be a basis of generalized real eigenvectors of A. If xp = yur a; V;, then x(t, x9) = 
ae: , 0 X(t, v;) for all t € IR. This reduces the computation of solutions to x = Ax to the compu- 
tation of solutions for Jordan blocks; see the examples below or [HSD04, Chap. 5] for a discussion 
of this topic. 

7. Each generalized real eigenspace E;, is invariant for the linear differential equation x = Ax, i.e., for 
Xo € E, it holds that x(t, xo) € Ex for allt €¢ R. 

8. The Lyapunov exponent A(x9) of a solution x(-,xo) (with x9 4 0) satisfies A(xo) = lim;+00 
+ log x(t, Xo) || = Aj; if and only if xo € L(Aj;). Hence, associated to a matrix A € gl(d,R) are 
exactly / Lyapunov exponents, the distinct real parts of the eigenvalues of A. 

9. The following are equivalent: 


(a) The zero solution x(t, 0) = 0 of the differential equation x = Ax is asymptotically stable. 
(b) The zero solution is exponentially stable 


(c) All Lyapunov exponents are negative. 


(d) L7 =R?4. 
Examples: 
1. Let A = diag(aj,...,aq) bea diagonal matrix. Then the solution of the linear differential equation 
ent 
x = Ax with initial value xp € R@ is given by x(t, xo) = e4'xy = . Xo. 


efat 


2. Lete; = (1,0,...,0)",...,e¢ = (0,0,..., 1)? bethe standard basis of R@. Then {x(-,e1),...,x(-,ea)} 
is a basis of the solution space sol(A). 

3. Let A = diag(a),...,aq) be a diagonal matrix. Then the standard basis {e),...,eg} of R? consists 
of eigenvectors of A. 
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. Let A € gl(d,R) be diagonalizable, i.e., there exists a transformation matrix T € GL(d,R) and 


a diagonal matrix D € gl(d,IR) with A = T~'DT. Then the solution of the linear differential 
equation x = Ax with initial value x) € R@ is given by x(t,xo) = T~!e?'T xo, where e” is given 
in Example 1. 

—v 


. Let B= : be the real Jordan block associated with a complex eigenvalue uw = A + iv of 


Xr 
the matrix A € gl(d,R). Let yo € E(A, 2), the real eigenspace of jz. Then the solution y(t, yo) 


sa oe 23 oe 4, |COSVE =— sin vt : 8 
of y = By is given by y(t, yo) = e se ee. edeae ENC! According to Fact 6 this is also the 


E (A, 4)-component of the solutions of x = J Ex. 


. Let B be a Jordan block of dimension n associated with the real eigenvalue yz of a matrix A € 


gl(d, IR). Then for 


B= one has e®! = e# 


1 : t 
LL 1 


In other words, for yo = [y1,---> Yn] T © E(A, 2), the jth component of the solution of y = By 
reads yj(t,yo) = e“" ie; tony According to Fact 6 this is also the E (A, jz)-component of ela", 


. Let B beareal Jordan block of dimension n = 2massociated with the complex eigenvalue jp = A+iv 


of a matrix A € gl(d,R). Then with D = ; a and [ = tt 


1 
pan can ra pos 
DiI DED oD 8 Gay 
B= — one has e?! = e* pa [> 

oe &D 
I $B 
D D 

= |cosvt —sinvt = Yr : 

where D = bee aaa i In other words, for yo = [715 21)---5Vm>Zm]” € E(A, 2), the jth 


components, j = 1,...,m, of the solution of y = By read 


k " 
a (ye Cos vt — z, sin vt), 


yj(tsyo) =e" SO 
ij 


m 
kj ‘ 
Z(t yo) =e" a ay (2k cos vt + yz sin vt). 
=j 


According to Fact 6, this is also the E (A, 4)-component of eat, 

Using these examples and Facts 5 and 6, it is possible to compute explicitly the solutions to any 
linear differential equation in R®. 

Recall that for any matrix A there is a matrix T € GL(d,R) such that A = T7! i T, where wT is 
the real Jordan canonical form of A. The exponential behavior of the solutions of x = Ax can be 
read off from the diagonal elements of J *. 
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56.2 Linear Dynamical Systems in R4 


The solutions of a linear differential equation x = Ax, where A € g/(d,R), define a (continuous time) 
dynamical system, or linear flow in R*. The standard concepts for comparison of dynamical systems 
are equivalences and conjugacies that map trajectories into trajectories. For linear flows in R@ these 
concepts lead to two different classifications of matrices, depending on the smoothness of the conjugacy 
or equivalence. 


Definitions: 


The real square matrix A is hyperbolic if it has no eigenvalues on the imaginary axis. 
A continuous dynamical system over the “time set” R with state space M, a complete metric space, is 
defined asa map ® : R x M —> M with the properties 


(i) ®(0,x) = x forall x € M, 
(ii) ®(s + t,x) = O(s, P(t, x)) foralls,t ¢ Randall x € M, 
(iii) ® is continuous (in both variables). 


The map © is also called a (continuous) flow. 

For each x € M the set {®(t, x), t € R} is called the orbit (or trajectory) of the system through x. 

For each t € R the time-t map is defined as y, = ®(t,-) : M —> M. Using time-t maps, the properties 
(i) and (ii) above can be restated as (i)’ gp = id, the identity map on M, (ii)’ @4+ = Y; oY foralls,t € R. 

A fixed point (or equilibrium) of a dynamical system ® is a point x € M with the property ®(t, x) = x 
forallt € R. 

An orbit {®(t,x),t € R} of a dynamical system © is called periodic if there exists t € R,f > 0 such 
that ®(f + s,x) = ®(s,x) for alls € R. The infimum of the positive f € R with this property is called 
the period of the orbit. Note that an orbit of period 0 is a fixed point. 

Denote by C K(X, Y) (k => 0) the set of k-times differentiable functions between C‘-manifolds X and 
Y, with C° denoting continuous. 

Let ®, Y : R x M —> M be two continuous dynamical systems of class C K(k > 0), ie., fork > 1 the 
state space M is at least a C‘-manifold and ®, V are Ck-maps. The flows ® and W are: 


(i) C k_ equivalent (k > 1) if there exists a (local) C-diffeomorphism h : M — M such that h takes 
orbits of ® onto orbits of V, preserving the orientation (but not necessarily parametrization by 
time), ie., 


(a) For each x € M there is a strictly increasing and continuous parametrization map Tt, : R > 
R such that h(®(t, x)) = W(t,(t), h(x)) or, equivalently, 


(b) Forallx € Mand6é > Othereexistse > Osuch that forallt € (0,5), h(®(t,x)) = W(t’, h(x)) 
for some t’ € (0,€). 


(ii) C*-conjugate (k > 1) if there exists a (local) C*-diffeomorphism h : M -—> M such that 
h(®(t,x)) = W(t, h(x)) for allx € Mandt eR. 


Similarly, the flows © and W are C°-equivalent if there exists a (local) homeomorphism h : M > M 
satisfying the properties of (i) above, and they are C°-conjugate if there exist a (local) homeomorphism 
h: M + Msatisfying the properties of (ii) above. Often, C°-equivalence is called topological equivalence, 
and C°-conjugacy is called topological conjugacy or simply conjugacy. 

Warning: While this terminology is standard in dynamical systems, the terms conjugate and equivalent 
are used differently in linear algebra. Conjugacy as used here is related to matrix similarity (cf. Fact 6), not 
to matrix conjugacy, and equivalence as used here is not related to matrix equivalence. 
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Facts: 
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Literature: [HSD04], [Rob98]. 


1. 
2. 


10. 


11. 


If the flows ® and W are C*-conjugate, then they are C*-equivalent. 

Each time-t map ¢; has an inverse (y,)~! = g_;, and gy : M —> M isa homeomorphism, i.e., a 
continuous bijective map with continuous inverse. 

Denote the set of time-t maps again by ® = {g;, t € R}. A dynamical system is a group in the sense 
that (®, 0), with o denoting composition of maps, satisfies the group axioms, and g : (R,+) —> 
(®, 0), defined by y(t) = ¢, is a group homomorphism. 

Let M be a C™-differentiable manifold and X a C~™-vector field on M such that the differential 
equation x = X(x) has unique solutions x(t, xo) for all x9 € M and allt € R, with x(0, x9) = xo. 
Then ®(t, xo) = x(t, xo) defines a dynamical system ® : R x M —> M. 


. A point x9 € M isa fixed point of the dynamical system ® associated with a differential equation 


x = X(x) as above if and only if X(xo) = 0. 
For two linear flows ® (associated with x = Ax) and W (associated with x = Bx) in R4, the 
following are equivalent: 


* ®and W are C-conjugate fork > 1. 
* @and W are linearly conjugate, i.e., the conjugacy map h isa linear operator in GL(R®). 


¢ Aand B are similar, i.e., A= TBT~! forsome T € GL(d,R). 


Each of the statements in Fact 6 implies that A and B have the same eigenvalue structure and 
(up to a linear transformation) the same generalized real eigenspace structure. In particular, the 
C*-conjugacy classes are exactly the real Jordan canonical form equivalence classes in gl(d, R). 
For two linear flows ® (associated with x = Ax) and W (associated with x = Bx) in R4, the 
following are equivalent: 


* ® and W are C* -equivalent for k > 1. 
* ® and W are linearly equivalent, i.e., the equivalence map h is a linear map in GL(R’). 


* A=aTBT™! for some positive real number w and T €¢ GL(d,R). 


Each of the statements in Fact 8 implies that A and B have the same real Jordan structure and 
their eigenvalues differ by a positive constant. Hence, the C k equivalence classes are real Jordan 
canonical form equivalence classes modulo a positive constant. 

The set of hyperbolic matrices is open and dense in gl(d, R). A matrix A is hyperbolic if and only 
if it is structurally stable in g/(d, R), ie., there exists a neighborhood U C gl(d,R) of A such that 
all B € U are topologically equivalent to A. 

If A and B are hyperbolic, then the associated linear flows ® and W in R@ are C°-equivalent (and 
C °_conjugate) if and only if the dimensions of the stable subspaces (and, hence, the dimensions of 
the unstable subspaces) of A and B agree. 


Examples: 


1. 


Linear differential equations: For A € gl(d,R) the solutions of x = Ax form a continuous 
dynamical system with time set R and state space M = R*: Here ® : R x R? —> R’ is defined 
by ®(t,x0) = x(t, x9) = e4*xo. 

Fixed points of linear differential equations: A point x) € R@ is a fixed point of the dynamical 
system ® associated with the linear differential equation x = Ax if and only if x9 € ker A, the 
kernel of A. 

Periodic orbits of linear differential equations: The orbit ®(f, x9) := x(t, Xo), t € Ris periodic with 
period f > 0 if and only if xp is in the eigenspace of a nonzero complex eigenvalue with zero real 
part. 
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4. For each matrix A € gl(d,R) its associated linear flow in R¢ is C*-conjugate (and, hence, 
C k equivalent) for all k > 0 to the dynamical system associated with the Jordan form J ie 


56.3 Chain Recurrence and Morse Decompositions of Dynamical 
Systems 


A matrix A € gl(d,R) and, hence, a linear differential equation x = Ax maps subspaces of R¢ into 
subspaces of R“. Therefore, the matrix A also defines dynamical systems on spaces of subspaces, such as 
the Grassmann and the flag manifolds. These are nonlinear systems, but they can be studied via linear 
algebra, and vice versa; the behavior of these systems allows for the investigation of certain properties of the 
matrix A. The key topological concepts for the analysis of systems on compact spaces like the Grassmann 
and flag manifolds are chain recurrence, Morse decompositions, and attractor—repeller decompositions. 
This section concentrates on the first two approaches, the connection to attractor—repeller decompositions 
can be found, e.g., in [CK00, App. B2]. 


Definitions: 


Given a dynamical system ® : R x M —> M, for a subset N C M the qa-limit set is defined as 
a(N) = {y € M, there exist sequences x, in N and t, — —oo in R with lim, ®(t,,x,) = y}, and 
similarly the w-limit set of N is defined as w(N) = {y € M, there exist sequences x, in N and t, — oo 
in R with lim, 40 ®(ti, Xn) = y}. 

For a flow ® on a complete metric space M and e, T > 0, an (e, T)-chain from x € Mtoy € Mis 
given by 


neN, Xp = X,..-,Xn = Vy, Toadies LA > T 
with 
d(®( Tj, x;), xj41) < € for alli, 


where d is the metric on M. 

Aset K C Mis chain transitive if for all x, y € K andalle, T > 0 there is an (e, T)-chain from x to y. 

The chain recurrent set CR is the set of all points that are chain reachable from themselves, i.e., 
CR = {x € M, for alle, T > 0 there is an (¢, T)-chain from x to x}. 

A set M C M isa chain recurrent component, if it is a maximal (with respect to set inclusion) chain 
transitive set. In this case M is a connected component of the chain recurrent set CR. 

For a flow ® on a complete metric space M, a compact subset K C M is called isolated invariant, if it 
is invariant and there exists a neighborhood N of K, i.e., a set N with K C int N, such that ®(t,x) € N 
for allt € Rimplies x € K. 

A Morsedecomposition ofa flow ® onacomplete metric space M isa finite collection{M;, i = 1,...,1} 
of nonvoid, pairwise disjoint, and isolated compact invariant sets such that 


1 
(i) For all x € M, w(x), a(x) C mis and 


i=1 


l 
(ii) Suppose there are Mj,,Mj,,...,Mj, and x1,...,x%, € M\ mi with a(x;) C Mj,_, and 
i=l 


o(x;) C Mj, fori =1,...,n; then Mj, A M;j,. 


The elements of a Morse decomposition are called Morse sets. 
A Morse decomposition {M,;, i = 1,...,/} is finer than another decomposition {Nj io eee n}, 
if for all MM; there exists an index j € {1,...,”} such that M; C Nj. 
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Facts: 
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Literature: [Rob98], [CK00], [ACK05]. 


L. 


For a Morse decomposition {Mj, i = 1,...,/} the relation M; < Mj, given by a(x) C Mj; and 
w(x) C M; for some x € M\UI_Mi, induces an order. 

Let &, W : R x M —> M be two dynamical systems on a state space M and leth: M > Mbea 
topological equivalence for ® and VW. Then 


(i) The point p € M isa fixed point of ® if and only if h(p) is a fixed point of W; 
(ii) The orbit ®(-, p) is closed if and only if Y(-, h(p)) is closed; 


(iii) If K C Mis ana-(or w-) limit set of ® from p € M, then h [K] is an a-(or w-) limit set of 
W from h(p) € M. 


(iv) Given, in addition, two dynamical systems ©). : R x N —> N,ifh: M— Misa topo- 
logical conjugacy for the flows ® and VY on M, andg : N > Nisa topological conjugacy for 
©, and ©, on N, then the product flows ® x ©, and VW x ©, on M x N are topologically 
conjugate viah x g: Mx N—> Mx N. This result is, in general, not true for topological 
equivalence. 


. Topological equivalences (and conjugacies) on a compact metric space M map chain transitive sets 


onto chain transitive sets. 

Topological equivalences map invariant sets onto invariant sets, and minimal closed invariant sets 
onto minimal closed invariant sets. 

Topological equivalences map Morse decompositions onto Morse decompositions. 


Examples: 


1. 


Dynamical systems in R!: Any limit set a(x) and w(x) from a single point x of a dynamical system 
in R! consists of a single fixed point. The chain recurrent components (and the finest Morse 
decomposition) consist of single fixed points or intervals of fixed points. Any Morse set consists of 
fixed points and intervals between them. 

Dynamical systems in R?: A nonempty, compact limit set of a dynamical system in R?, which 
contains no fixed points, is a closed, i.e., a periodic orbit (Poincaré—Bendixson). Any nonempty, 
compact limit set of a dynamical system in R? consists of fixed points, connecting orbits (such as 
homoclinic or heteroclinic orbits), and periodic orbits. 


. Consider the following dynamical system ® in R?\{0}, given by a differential equation in polar 


form forr > 0,0 € [0,27), anda # 0: 
r=1—7, O=a. 


For each x € R*\ {0} the w-limit set is the circle w(x) = S' = {(r,0),r = 1,6 € [0, 27r)}. The state 
space R*\{0} is not compact, and a-limit sets exist only for y € S', for which a(y) = S!. 
Consider the flow ® from the previous example and a second system W, given by 


f=1—-r, O=b 


with b # 0. Then the flows ® and W are topologically equivalent, but not conjugate if b a. 

An example ofa flow for which the limit sets from points are strictly contained in the chain recurrent 
components can be obtained as follows: Let M = [0,1] x [0,1]. Let the flow ® on M be defined 
such that all points on the boundary are fixed points, and the orbits for points (x, y) € (0, 1) x (0, 1) 
are straight lines ®(-, (x, y)) = {(Z1, 22), Z1 = X, Z2 € (0, 1)} with lim;_,+.. P(t, (x, y)) = (x, +1). 
For this system, each point on the boundary is its own a- and w-limit set. The w-limit sets for points 
in the interior (x, y) € (0,1) x (0,1) are of the form {(x, —1)}, and the w-limit sets are {(x,+1)}. 
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The only chain recurrent component for this system is M = [0,1] x [0, 1], which is also the only 
Morse set. 


56.4 Linear Systems on Grassmannian and Flag Manifolds 


Definitions: 


The kth Grassmannian G, of R@ can be defined via the following construction: Let F (k, d) be the set of 
k-frames in R@, where a k-frame is an ordered set of k linearly independent vectors in IR‘. Two k-frames 
X = [xj,...,x;] and Y = [y),...,y«] are said to be equivalent, X ~ Y, if there exists T € GL(k, R) 
with X" = TY’, where X and Y are interpreted as d x k matrices. The quotient space G; = F (k,d)/ ~ 
is a compact, k(d — k)-dimensional differentiable manifold. For k = 1, we obtain the projective space 
P4! —G, inR’. 

The kth flag of R’ is given by the following k—sequences of subspace inclusions, 


Fy = {Fy = (Vi,...5 Ve)s Vi C Vin and dim V; = 7 for alli}. 


For k = d, this is the complete flag F = Fy. 

Each matrix A € gl(d, R) defines a map on the subspaces of R? as follows: Let V = Span({x,,...,Xx}). 
Then AV = Span({Ax;,..., Axg}). 

Denote by G, ® and F;.® the induced flows on the Grassmannians and the flags, respectively. 


Facts: 
Literature: [Rob98], [CK00], [ACK05]. 


1. Let P® be the projection onto P¢—! ofa linear flow ®(t, x) = e4"x. Then P@ has / chain recurrent 
components {M,,...,.M_}, where / is the number of different Lyapunov exponents (i.e., of differ- 
ent real parts of eigenvalues) of A. For each Lyapunov exponent A;, M; = PL;, the projection of 
the ith Lyapunov space onto P?-!, Furthermore {M,,...,4)} defines the finest Morse decom- 
position of P® and M; ~ M, if and only if A; < A;. 

2. For A, B € gl(d,R), let P® and PW be the associated flows on P4-! and suppose that there is 
a topological equivalence h of P® and PW. Then the chain recurrent components Nj,...,NV; 
of PW are of the form N; = h[M;], where M; is a chain recurrent component of P®. In 
particular, the number of chain recurrent components of P® and PW agree, and h maps the order 
on {M,,...,M)} onto the order on {N4,...,Nj}. 

3. For A, B € gl(d,R) let P® and PW be the associated flows on P4—! and suppose that there is 
a topological equivalence h of P® and PW. Then the projected subspaces corresponding to real 
Jordan blocks of A are mapped onto projected subspaces corresponding to real Jordan blocks of 
B preserving the dimensions. Furthermore, h maps projected eigenspaces corresponding to real 
eigenvalues and to pairs of complex eigenvalues onto projected eigenspaces of the same type. This 
result shows that while C°-equivalence of projected linear flows on P¢~! determines the number! of 
distinct Lyapunov exponents, it also characterizes the Jordan structure within each Lyapunov space 
(but, obviously, not the size of the Lyapunov exponents nor their sign). It imposes very restrictive 
conditions on the eigenvalues and the Jordan structure. Therefore, C °_equivalences are not a useful 
tool to characterize /. The requirement of mapping orbits into orbits is too strong. A weakening 
leads to the following characterization. 

4. Two matrices A and B in gl(d, R) have the same vector of the dimensions d; of the Lyapunov spaces 
(in the natural order of their Lyapunov exponents) if and only if there exist a homeomorphism h : 
pé-! _, P4—! that maps the finest Morse decomposition of P® onto the finest Morse decomposition 
of PW, i.e., h maps Morse sets onto Morse sets and preserves their orders. 


56-10 Handbook of Linear Algebra 


5. Let A € gl(d,R) with associated flows ® on R¢ and F,® on the k-flag. 


(i) For every k € {1,...,d} there exists a unique finest Morse decomposition {Mi} of F,®, 
where i; € {1,...,d yk is a multi-index, and the number of chain transitive components in 
F; is bounded by ree 


(ii) Let M; withi € {1,...,d}* beachain recurrent component in F;,_,. Consider the (d —k+1)- 
dimensional vector bundle z : W(M;) > M; with fibers 


W(Mi) my, = R4/ Vi-1 for Fy = (Vis---5 Ven) © Mi C Fro. 


Then every chain recurrent component pMi, >j=l,...,k; <d—k +1, of the projective 
bundle PW(M,;) determines a chain recurrent component ,.M; ; on; via 


Mi, = { Fr = (Frau, Ve) € Fes Fee € Mj and P(Vk/ Vi-1) € pMi; } ; 


Every chain recurrent component in F; is of this form; this determines the multiindex i; 
inductively for k = 2,...,d. 


6. Onevery Grassmannian G; there exists a finest Morse decomposition of the dynamical system G;®. 
Its Morse sets are given by the projection of the chain recurrent components from the complete 
flag F. 

7. Let A € gl(d,R) be a matrix with flow ® on R‘. Let L;,i = 1,...,1, be the Lyapunov spaces of 
A, Le., their projections PL; = M, are the finest Morse decomposition of P® on the projective 
space. For k = 1,...,d define the index set 


T(k) = {(ki,...5kim) thy +... +km =k and0 <k; < d; =dimL;}. 
Then the finest Morse decomposition on the Grassmannian G;, is given by the sets 
NE og, = Gili @..... B Ge, Lm (kis. +skm) € IK). 


8. For two matrices A, B € gl(d,R) the vector of the dimensions d; of the Lyapunov spaces (in the 
natural order of their Lyapunov exponents) are identical if and only if certain graphs defined on 
the Grassmannians are isomorphic; see [ACK05]. 


Examples: 


1. For A € gl(d,R) let ® be its linear flow in R¢. The flow ® projects onto a flow P® on pal, given 
by the differential equation 


$ =h(s, A) =(A—s'As I)s, withs € P4!. 
Consider the matrices 
A = diag(—1,—1, 1) and B = diag(—1,1, 1). 


We obtain the following structure for the finest Morse decompositions on the Grassmannians 
for A: 

G: My, ={Span(e,e2)} and. M3; ={Span(e;)} 

Gi: M42 ={Span(ej, e2)} and. M13 = {{Span(x, e3)} : x eSpan(ey, e)} 

G3: M423 ={Span(er, er, e3)} 

and for B we have 

G: NM, ={Span(e,)} and V5 ={Span(ep, e3) } 

Gz: Miz = {Span(e;,x) : x €Span(e,e;)} and N33 ={Span(e2, e3)} 

G3: Ni23 ={Span(ei, e2, e3)}. 
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On the other hand, the Morse sets in the full flag are given for A and B by 


M 1,2,3 M 1,2,3 M 1,2,3 Ni23 N23 N23 
Mi <x<|M re |S M3 and Ni2 x Ni2 x N23 > 
M, M M3 NM Ny Ny 


respectively. Thus, in the full flag, the numbers and the orders of the Morse sets coincide, while 
on the Grassmannians (together with the projection relations between different Grassmannians) 
one can distinguish also the dimensions of the corresponding Lyapunov spaces. (See [ACK05] for 
a precise statement.) 


56.5 Linear Skew Product Flows 


Developing a linear algebra for time varying systems x = A(t)x means defining appropriate concepts to 
generalize eigenvalues, linear eigenspaces and their dimensions, and certain normal forms that characterize 
the behavior of the solutions of a time varying system and that reduce to the constant matrix case if 
A(t) = A € gl(d,R). The eigenvalues and eigenspaces of the family {A(t), t € IR} do not provide an 
appropriate generalization; see, e.g., [Hah67], Chapter 62. For certain classes of time varying systems 
it turns out that the Lyapunov exponents and Lyapunov spaces introduced in section 56.1 capture the 
key properties of (real parts of) eigenvalues and of the associated subspace decomposition of R*. These 
systems are linear skew product flows for which the base is a (nonlinear) system 6, that enters into the linear 
dynamics of a differential equation in the form x = A(@,)x. Examples for this type of systems include 
periodic and almost periodic differential equations, random differential equations, systems over ergodic 
or chain recurrent bases, linear robust systems, and bilinear control systems. This section concentrates on 
periodic linear differential equations, random linear dynamical systems, and robust linear systems. It is 
written to emphasize the correspondences between the linear algebra in Section 56.1, Floquet theory, the 
multiplicative ergodic theorem, and the Morse spectrum and Selgrade’s theorem. 

Literature: [Arn98], [BK94], [CK00], [Con97], [Rob98]. 


Definitions: 


A (continuous time) linear skew-product flow is a dynamical system with state space M = Q x R¢ and 
flow®:RxQxR? —> Qx R4, where ® = (8, g) is defined as follows: 6 : R x Q —> Qisa dynamical 
system, andg:R x Q x R? —> R’ is linear in its R¢-component, ie., for each (t,w) € R x Q the map 
g(t,w,-) : R4 —> R? is linear. Skew-product flows are called measurable (continuous, differentiable) 
if Q = (6,q@) is a measurable space (topological space, differentiable manifold) and ® is measurable 
(continuous, differentiable). For the time-t maps, the notation 6, = O(t,-) : Q —> Q is used again. 

Note that the base component @ : R x Q —>» Qisadynamical system itself, while the skew-component 
¢g is not a dynamical system. The skew-component ¢ is often called a co-cycle over 0. 

Let ®:Rx Qx R4 — Q~x R¢ bea linear skew-product flow. For xp € R4, xo # 0, the Lyapunov 
exponent is defined as A(xo, w) = lim sup,_, t log ||p(t, @, Xo) ||, where log denotes the natural logarithm 
and || - || is any norm in R¢. 


Examples: 


1. Time varying linear differential equations: Let A : R —> gl(d,R) bea uniformly continuous func- 
tion and consider the linear differential equation x(t) = A(t)x(t). The solutions of this differential 
equation define a dynamical system via ® : R x R x R4 —> R x R4, where 9: R x R — Ris 
given by O(t,t) =t+t,andg:RxRx IR? —> R‘ is defined as y(t, T, Xo) = X(t+T, T)Xo. Here 
X(t,T) is a fundamental matrix of the differential equation X(t) = A(t)X(t) in gl(d,R). Note 
that for y(t, t,-) : R4 —> R4, t € R, we have g(t +s,t) = y(t, 6(s,T)) 0 (s, T) and, hence, the 
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solutions of x(t) = A(t)x(t) themselves do not define a flow. The additional component 6 “keeps 
track of time.” 

2. Metric dynamical systems: Let (Q,¥, P) be a probability space, ie., a set Q with o-algebra F and 
probability measure P. Let 6 : R x Q —> Q bea measurable flow such that the probability 
measure P is invariant under 0, i.e., 0;P = P for all t € R, where for all measurable sets X € F 
we define 6,P(X) = P{@7!(X)} = P(X). Flows of this form are often called metric dynamical 
systems. 

3. Random linear dynamical systems: A random linear dynamical system is a skew-product flow 
®:RxQ~x Rt —> Q x R4, where (Q,F, P,6) is a metric dynamical system and each 
gy: Rx Q x R¢ —> R‘ is linear in its R¢-component. Examples for random linear dynamical 
systems are given, e.g., by linear stochastic differential equations or linear differential equations 
with stationary background noise; see [Arn98]. 

4. Robust linear systems: Consider a linear system with time varying perturbations of the form 
& = A(u(t))x = Aox + D2", u(t) Aix, where Ao,...,Am € gl(d,R), u EU = {u: R > 
U, integrable on every bounded interval}, and U C R” is compact, convex with 0 ¢€ int U. 
A robust linear system defines a linear skew-product flow via the following construction: We 
endow U with the weak*-topology of L°(R,U)* to make it a compact, metrizable space. The 
base component is defined as the shift 9 : R x U —> U, O(t,u(-)) = u(- + t), and the skew- 
component consists of the solutions g(t, u(-),x), t € R of the perturbed differential equation. 
Then ®: Rx U x R4 — U x R4, P(t, u,x) = (A(t, u), v(t, u,x)) defines a continuous linear 
skew-product flow. The functions u can also be considered as (open loop) controls. 


56.6 Periodic Linear Differential Equations: Floquet Theory 


Definitions: 

A periodic linear differential equation x = A(6,)x is given by a matrix function A : R —> gl(d,R) 
that is continuous and periodic (of period f > 0). As above, the solutions define a dynamical system via 
®:RxS! x R? —> S! x R4, if we identify R modf with the circle S!. 


Facts: 
Literature: [Ama90], [GH83], [Hah67], [Sto92], [Wig96]. 


1. Consider the periodic linear differential equation x = A(@,)x with period f > 0. A fundamental 
matrix X(t) of the system is of the form X(t) = P(t)e® for t € R, where P(-) isa nonsingular, 
differentiable, and t-periodic matrix function and R € gl(d,C). 


2. Let X(-) be a fundamental solution with X(0) = I € GL(d,R). The matrix X(t) = e®* is called 
the monodromy matrix of the system. Note that R is, in general, not uniquely determined by X, 
and does not necessarily have real entries. The eigenvalues w;, j = 1,...,d of X (f) are called the 
characteristic multipliers of the system, and the eigenvalues yz; = 4;-+iv; of R are the characteristic 


2mm 


exponents. It holds that 4; = *log aro; j =1,...,dandm € Z. This determines uniquely 


the real parts of the characteristic exponents A; = Re 1; = log | jj) J = 1,...,d. The A; are 
called the Floquet exponents of the system. 

3. Let b = (6,9): Rx S! x R? —> S! x R“ be the flow associated with a periodic linear differential 
equation x = A(f)x. The system has a finite number of Lyapunov exponents Aj, j = 1,...,] < d. 


I L(Aj,T) of R¢ into 


For each exponent A; and each t € S! there exists a splitting R¢ = QB 
linear subspaces with the following properties: 


(a) The subspaces L(A ;, tT) have the same dimension independent of t, i.e., foreach j = 1,...,1 
it holds that dim L(A;,0) = dim L(Aj;,t) =: d; for allo,t € sl. 
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(b) The subspaces L (Aj, 7) are invariant under the flow ®, ie., foreach j = 1,...,/ it holds that 
p(t, T)L(A;,T) = L(Aj, (ft, 7)) = L(Aj,t+T) for allt €¢ Randt € Ss). 


(c) A(x, T) = lim;-++450 + log le(t, t,x) || = A; ifand only ifx € L(Aj;,7)\{0}. 


4. The Lyapunov exponents of the system are exactly the Floquet exponents. The linear subspaces 
L(Aj,-) are called the Lyapunov spaces (or sometimes the Floquet spaces) of the periodic matrix 
function A(t). 

5. For each j = 1,...,1 < d the map L; :Si — Ga, defined by t +> L(Aj,T) is continuous. 

6. These facts show that for periodic matrix functions A : R —> gl(d,R) the Floquet exponents 
and Floquet spaces replace the real parts of eigenvalues and the Lyapunov spaces, concepts that 
are so useful in the linear algebra of (constant) matrices A € gl(d,R). The number of Lyapunov 
exponents and the dimensions of the Lyapunov spaces are constant for t € S!, while the Lyapunov 
spaces themselves depend on the time parameter t of the periodic matrix function A(t), and they 
form periodic orbits in the Grassmannians Gy, and in the corresponding flag. 

7. As an application of these results, consider the problem of stability of the zero solution of x(t) = 
A(t)x(t) with periodt > 0: The stable, center, and unstable subspaces associated with the periodic 
matrix function A : R —> gl(d,R) are defined as L~(t) = QP{L(A;,7), Aj < O}, L(t) = 
Q{L(A;,7), Aj = O}, and Lt(r) = @{L(Aj,7), Aj > O}, respectively, for t € S!. The zero 
solution x(t, 0) = 0 of the periodic linear differential equation x = A(t)x is asymptotically stable 
if and only if it is exponentially stable if and only if all Lyapunov exponents are negative if and only 
if L~(t) = R¢ for some (and hence for all) t € S!. 

8. Another approach to the study of time-dependent linear differential equations is via transforming 
an equation with bounded coefficients into an equation of known type, such as equations with 
constant coefficients. Such transformations are known as Lyapunov transformations; see [Hah67, 
Secs. 61-63]. 


Examples: 


1. Consider the t-periodic differential equation x = A(t)x. This equation has a nontrivial t-periodic 
solution iff the system has a characteristic multiplier equal to 1; see Example 2.3 for the case with 
constant coefficients ([Ama90, Prop. 20.12]). 

2. Let H be a continuous quadratic form in 2d variables x),...,xg,V1,...,¥q and consider the 
Hamiltonian system 


= i= of el ee ce 
OY; A Ox; 
Uaneot = eet = = Ay Ap]. 
gz = [x’,y |], wecan set H(x,y,t) =z A(t)z, where A = r oA with A;, and Az 
12 22 


symmetric, and, hence, the equation takes the form 


AL(t) A(t) 
ys 


= z=: P(t)z. 
—Aj,(t) a 


0 
Note that — P7(t) = QP(t)Q7! with Q = f F jv Tis the d x d identity matrix. Assume 


that H is t-periodic, then the equation for z and its adjoint have the same Floquet exponents and 
for each exponent A its negative —A is also a Floquet exponent. Hence, the fixed point 0 € R*@ 
cannot be exponentially stable ({Hah67, Sec. 60]). 

3. Consider the periodic linear oscillator 


ytat)y + qty =0. 
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Using the substitution y = z exp(— ; {qi(u)du) one obtains Hill’s differential equation 


Z+ p(t)z = 0, p(t) := qo(t) 


Its characteristic equation is 4” — 2a +1 = 0, with a still to be determined. The multipliers satisfy 
the relations a}a@2 = 1 and a + a = 2a. The exponential stability of the system can be analyzed 
using the parameter a: If a” > 1, then one of the multipliers has absolute value > 1 and, hence, the 
system has an unbounded solution. If a? = 1, then the system has a nontrivial periodic solution 
according to Example 1. Ifa? < 1, then the system is stable. The parameter a can often be expressed 
in form of a power series; see [Hah67, Sec. 62] for more details. A special case of Hill’s equation is 
the Mathieu equation 


Z+ (B, + Bo cos2t)z = 0, 


with 6;, 6 real parameters. For this equation numerically computed stability diagrams are available; 
see [Sto92, Secs. VI. 3 and 4]. 


56.7 Random Linear Dynamical Systems 


Definitions: 


Let 6: R x Q —> Q bea metric dynamical system on the probability space (Q,F, P). Aset A € F is 
called P-invariant under 6 if P[(@~!(t, A) \\ A)U(A\ 671(t, A))] = 0 for all t € R. The flow 6 is called 
ergodic, if each invariant set A € F has P-measure 0 or 1. 


Facts: 
Literature: [Arn98], [Con97]. 


1. (Oseledets Theorem, Multiplicative Ergodic Theorem) Consider a random linear dynamical system 
® = (6,9): Rx Qx R? —> Q x R4 and assume 


sup _log* |lp(t,)ll € L'(Q2,F,P) and sup _log* ||y(t,@)'|| € L'(2,F,P), 
0<t<1 O<t<l 
where || - || is any norm on GL(d,R), L' is the space of integrable functions, and log* denotes the 
positive part of log, i.e., 


ip es log(x) for log(x) > 0 
0 for log(x) < 0. 


Then there exists a set 2 C Q of full P -measure, invariant under the flow @ : R x Q —> Q, such 
that for each w € Q there is a splitting R4? = Bi" L ;(@) of R¢ into linear subspaces with the 
following properties: 


(a) The number of subspaces is 9-invariant, ie., 1(O(t, @)) = 1(w) for all t € R, and the dimen- 
sions of the subspaces are @-invariant, i.e., dim L ;(0(t,w)) = dim L ;(@) =: d;(@) for all 
teR. 

(b) The subspaces are invariant under the flow ®, i.e., o(t,@)L j(@) C Lj(@(t,@)) for all j = 
1,...,1(@). 

(c) There exist finitely many numbers A;(@) < ... < Ayw)(@) in R (with possibly 41(@) = 
—oo), such that for each x € R@\{0} the Lyapunov exponent A(x, w) exists as a limit and 
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A(x, w) = lim; +00 + log le(t, t,x) || = Aj (@) if and only ifx € L j(@)\{0}. The subspaces 
L ;(@) are called the Lyapunov (or sometimes the Oseledets) spaces of the system ®. 


2. The following maps are measurable: 1 : 2 —> {1,...,d} with the discrete o-algebra, and for each 
j =1,...,1(@) the maps L; : 2 — Ga, with the Borel o-algebra, dj : 2 —> {1,...,d} with 
the discrete o-algebra, and A; : 2 —> RU {—oo} with the (extended) Borel o-algebra._ 

3. Ifthe base flow @ : R x 2 —> Qis ergodic, then the maps /, d;, and A; are constant on Q, but the 
Lyapunov spaces L ;(w) still depend (in a measurable way) on w € Q. 

4. As an application of these results, we consider random linear differential equations: Let (I, €, Q) 
bea probability space and € : R x T —> R" astochastic process with continuous trajectories, i.e., 
the functions &(-, y) : R —> R” are continuous for all y € I’. The process & can be written as a 
measurable dynamical system in the following way: Define Q = C(R, R”), the space of continuous 
functions from R to R”. We denote by F the o-algebra on Q generated by the cylinder sets, i.e., 
by sets of the form Z = {w € Q, w(t) € Fi,...,@(ti) € Fy, n € N, F; Borel sets in R”}. The 
process € induces a probability measure P on (Q,F) via P(Z) = Q{y €T, &(tj,y) € F; for 
i = 1,...,n}. Define the shift @ : R x Q —> R x Qas O(t,@(-)) = w(t + -). Then (Q,F, P,@) 
is a measurable dynamical system. If & is stationary, ie., if for all n € N, and t,t,...,t, € R, and 
all Borel sets F),..., F, in R”, it holds that Q{fy € T, &(t;,y) € F; fori = 1,...,n} = Qfy ET, 
&(t; +t, y) € F; fori = 1,...,n}, then the shift 6 on Q is P-invariant, and (Q, F, P,@) isa metric 
dynamical system. 

5. Let A : Q —~+ gl(d,R) be measurable with A € L!. Consider the random linear differential 
equation x(t) = A(6(t,@))x(t), where (Q,F, P,@) is a metric dynamical system as described 
before. We understand the solutions of this equation to be w-wise. Then the solutions define a 
random linear dynamical system. Since we assume that A € £', this system satisfies the integrability 
conditions of the Multiplicative Ergodic Theorem. 

6. Hence, for random linear differential equations x(t) = A(6(t,w))x(t) the Lyapunov exponents 
and the associated Oseledets spaces replace the real parts of eigenvalues and the Lyapunov spaces 
of constant matrices A € g/(d,R). If the “background” process & is ergodic, then all the quantities 
in the Multiplicative Ergodic Theorem are constant, except for the Lyapunov spaces that do, in 
general, depend on chance. 

7. The problem of stability of the zero solution of x(f) = A(@(t,@))x(t) can now be analyzed in 
analogy to the case ofa constant matrix or a periodic matrix function: The stable, center, and unstable 
subspaces associated with the random matrix process A(6(t, w)) are defined as L~(w) = G{L j(@), 
Aj (@) < 0}, L°(w) = Q{L j(w), 4; (@) = 0}, and Lt (w) = G{L j(w), Aj(w) > O}, respectively 
for wm € Q. We obtain the following characterization of stability: The zero solution x(t, w,0) = 0 
of the random linear differential equation x(t) = A(6(t, w))x(t) is P-almost surely exponentially 
stable if and only if P-almost surely all Lyapunov exponents are negative if and only if P{w € Q, 
L7(@) =R4}=1. 


Examples: 


1. The case of constant matrices: Let A € gl(d, R) and consider the dynamical system g : Rx R4 —> 
R¢ generated by the solutions of the linear differential equation x = Ax. The flow g can be 
considered as the skew-component of a random linear dynamical system over the base flow given 
by Q = {0}, F the trivial o-algebra, P the Dirac measure at {0}, and @ : R x Q —> Q defined as 
the constant map 0(t, w) = o for all t € R. Since the flow is ergodic and satisfies the integrability 
condition, we can recover all the results on Lyapunov exponents and Lyapunov spaces for g from 
the Multiplicative Ergodic Theorem. 

2. Weak Floquet theory: Let A : R —> gl(d,R) beacontinuous, periodic matrix function. Define the 
base flowas follows: 2 = S', Bis the Borel o -algebra onS', P is the uniform distribution on S', and 
0 is the shift 6(t, 7) = t+ 7. Then (Q, F, P,@) isan ergodic metric dynamical system. The solutions 
y(-,T,x) of x = A(t)x define a random linear dynamical system ® : R x Q x R? — Qx R¢ via 
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®(t,w,x) = (O(t,w), p(t, @,x)). With this set-up, the Multiplicative Ergodic Theorem recovers 
the results of Floquet Theory with P-probability 1. 

3. Average Lyapunov exponent: In general, Lyapunov exponents for random linear systems are difficult 
to compute explicitly —— numerical methods are usually the way to go. In the ergodic case, the average 
Lyapunov exponent A := 45> djA; is given byX = 4trE(A |Z), where A: 2 —> gl(d,R) is the 
random matrix of the system, and E (-, Z) is the conditional expectation of the probability measure 
P given the o-algebra Z of invariant sets on (2. As an example, consider the linear oscillator with 
random restoring force 


P(t) + 2By(t) + (L+ of @(6@))) y(t) = 0, 


where 6,0 € R are positive constants and f : Q > R is in L'. We assume that the background 
process is ergodic. Using the notation x; = y and x) = y we can write the equation as 


0 1 
x(t) = A(O(t, w)x(t) = -1-of(6lt,o)) -26 x(t). 


For this system we obtain 2 = —f ([Arn98, Remark 3.3.12]). 


56.8 Robust Linear Systems 


Definitions: 


Let db: RxU xR? —> Ux R? bealinear skew-product flow with continuous base flow@ : RxU —> U. 
Throughout this section, / is compact and @ is chain recurrent on U/. Denote by U/ x P4~! the projective 
bundle and recall that & induces a dynamical system P® : R x U x P*-! —> U x P*!. Fore, T > 0 
an (€, T)-chain ¢ of P® is given by n € N, 7o,...,T, => T, and (uo, po),..-5 (Uns pn) € U x P4-! with 
d(P®(T;,, uj, pi), View pici)) < ¢ fori =0,...,n-1. 

Define the finite time exponential growth rate of such a chain ¢ (or chain exponent) by 


n—1 


n—1 +t 
Ag) = i ‘) S~ (log ll p(Tis xi us)|l — log llxill), 
i=0 


i=0 


where x; € P~!(p;). 
Let M CU x P4! bea chain recurrent component of the flow P®. Define the Morse spectrum over 
Mas 


= Mo(M) = 


2 ER, there exist sequences €, > 0, T, — oo and 
(€n, T,)-chains ¢, in M such that limA(¢,) = a 


and the Morse spectrum of the flow as 


Sare(®) A € R, there exist sequences €, > 0, T, — oo and (é,, T,)- 
chains ¢,, in the chain recurrent set of P® such that limA(¢,) = A , 


Define the Lyapunov spectrum over M/ as 
Upy(M) = {A(u, x), (ux) € M, x F 0} 
and the Lyapunov spectrum of the flow ® as 


Lry(®) = {A(u,x), (ux) €U x R4,x F 0}. 
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Facts: 
Literature: [CK00], [Gru96], [HP05]. 


1. The projected flow P® has a finite number of chain-recurrent components M,,...,Mj,1 < d. 
These components form the finest Morse decomposition for P®, and they are linearly ordered 
M, <... < M). Their lifts P-'M; C U x R? form a continuous subbundle decomposition of 
Ux R4 = @j_, PM. 

2. The Lyapunov spectrum and the Morse spectrum are defined on the Morse sets, i.e., Upy(®) = 
Ujer Z1y(Mzj) and Zyo(®) = Uj, Eamo(Mi).- 

3. For each Morse set M; the Lyapunov spectrum is contained in the Morse spectrum, i.e., £1 ,(Mj) C 
Xmo(M;) fori = 1,..., 1. 

4. For each Morse set, its Morse spectrum is a closed, bounded interval Dyjo(M;) = [k},«;], and 
K*,Ki € Dry(M) fori = 1,...,1. 

5. The intervals of the Morse spectrum are ordered according to the order of the Morse sets, i.e., 
Mj < M; is equivalent to x < KF and kj < kj. 

6. Asan application of these results, consider robust linear systems of the form ® : R x U x R4 —> 
U x R4, given by a perturbed linear differential equation x = A(u(t))x := Aox+ aa 1 Ui (t) Aix, 
with Ao,..., Am € gl(d,R), u € U = {u: R —> U, integrable on every bounded interval} and 
U C R” is compact, convex with 0 € intU. Explicit equations for the induced perturbed system on 
the projective space P4~! can be obtained as follows: Let S‘~! C R¢ be the unit sphere embedded 
into R4. The projected system on S*~! is given by 


§(t) = h(u(t),s(t)), ue, s € S47! 


where 

h(u,s) = ho(s) + ew s) with h;(s) = (Ai ;—sTAjs-I)s,i=0,1,...,m. 
Define an equivalence relation on S4—! vias; ~ sy ifs; = —s2, identifying opposite points. Then the 
projective space can be identified as P4~! = S4~!/ ~. Since h(u,s) = —h(u, —s), the differential 


equation also describes the projected system on P?—!. For the Lyapunov exponents one obtains in 
the same way 


t 
A(u, x) = lim sup - log ||x(t) || = lim sup - | q(u(t),s(t)) dt 
0 


t>oo t>oo 


with 


q(u,s) = qo(s) + S_ uigi(s) with qi(s) — (Ai —sTAjs- 1) s, i=0,1,...,m. 


i=1 


For a constant perturbation u(t) = u € R for all t € R the corresponding Lyapunov exponents 
A(u, x) of the flow ® are the real parts of the eigenvalues of the matrix A(u) and the corresponding 
Lyapunov spaces are contained in the subbundles P~'/;. Similarly, if a perturbation u € U is 
periodic, the Floquet exponents of x = A(u(-))x are part of the Lyapunov (and, hence, of the 
Morse) spectrum of the flow ®, and the Floquet spaces are contained in P~'M,;. The systems 
treated in this example can also be considered as “bilinear control systems” and studied relative to 
their control behavior and (exponential) stabilizability — this is the point of view taken in [CKO0O]. 

7. For robust linear systems “generically” the Lyapunov spectrum and the Morse spectrum agree see 
[CKO00] for a precise definition of “generic” in this context. 

8. Of particular interest is the upper spectral interval Uyyo(M7) = [k;‘,«], as it determines the 
robust stability of kx = A(u(t))x (and stabilizability of the system if the set U/ is interpreted as a 
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set of admissible control functions; see [Gru96]). The stable, center, and unstable subbundles of 
U x R¢ associated with the perturbed linear system x = A(u(t))xare defined as L~ = @{P-'M pp 
kj < 0}, L°= @{P'M;,0€ [x,«;]}, and L+=Q{P'M;, k= > O}, respectively. The zero 
solution of x = A(u(t))x is exponentially stable for all perturbations u € U/ if and only if k; < 0 if 


and only if L~> =U x R4. 


Examples: 


1. In general, it is not possible to compute the Morse spectrum and the associated subbundle decom- 
positions explicitly, even for relatively simple systems, and one has to revert to numerical algorithms; 
compare [CK00, App. D]. Let us consider, e.g., the linear oscillator with uncertain restoring force 


Poke alee df 


with u(t) € [—p, p] and b > 0. Figure 56.1 shows the spectral intervals for this system depending 
on ep = 0. 

2. We consider robust linear systems as described in Fact 6, with varying perturbation range by 
introducing the family U? = pU for p > 0. The resulting family of systems is 


XP = A(u?(t))x? == Aox? + S > uP (1) Aix’, 


i=l 


with u? € U? = {u : R —> U®, integrable on every bounded interval}. The corresponding 
maximal spectral value «;(e) is continuous in p and we define the (asymptotic) stability radius of 
this family as r = inf{p > 0, there exists ug € U/? such that x? = A(uo(t))x? is not exponentially 
stable}. This stability radius is based on asymptotic stability under all time varying perturbations. 
Similarly one can introduce stability radii based on time invariant perturbations (with values in 
R” or C”) or on quadratic Lyapunov functions ([CK00], Chapter 11 and [HP05]). 

3. Linear oscillator with uncertain damping: Consider the oscillator 


yt+2b+ult)y+U+c)y =0 


-4 
0.0 0.5 1.0 1.5 2.0 2.5 


FIGURE 56.1 Spectral intervals depending on p > 0 for the system in Example 1. 
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t : + 
0.0 0.405 0.5 0.707 b 1.0 


FIGURE 56.2 Stability radii for the system in Example 4. 


with u(t) € [—p,p] andc € R. In equivalent first-order form the system reads 


x] _ 0 1 xX) (t) 0 0 Xx] 
gol |-l—c —2b| |x ve 0 2) lxo|° 


Clearly, the system is not exponentially stable for c < —1 with p = 0, and forc > —1 with p > b. 
It turns out that the stability radius for this system is 


0 for c<-—l 
r(c)= 


b for c>-—l. 


4. Linear oscillator with uncertain restoring force: Here we look again at a system of the form 


AoE alee JE 


with u(t) € [—p,p] andb > 0. (For b < 0 the system is unstable even for constant perturbations.) 
A closed form expression of the stability radius for this system is not available and one has to use 
numerical methods for the computation of (maximal) Lyapunov exponents (or maxima of the 
Morse spectrum); compare [CK00, App. D]. Figure 56.2 shows the (asymptotic) stability radius r, 
the stability radius under constant real perturbations rp, and the stability radius based on quadratic 
Lyapunov functions rz, all in dependence on b > 0; see [CKO0, Ex. 11.1.12]. 


56.9 Linearization 


The local behavior of the dynamical system induced by a nonlinear differential equation can be studied via 
the linearization of the flow. At a fixed point of the nonlinear system the linearization is just a linear differ- 
ential equation as studied in Sections 56.1 to 56.4. If the linearized system is hyperbolic, then the theorem 
of Hartman and Grobman states that the nonlinear flow is topologically conjugate to the linear flow. The 
invariant manifold theorem deals with those solutions of the nonlinear equation that are asymptotically 
attracted to (or repelled from) a fixed point. Basically these solutions live on manifolds that are described 
by nonlinear changes of coordinates of the linear stable (and unstable) subspaces. 

Fact 4 below describes the simplest form of the invariant manifold theorem at a fixed point. It can 
be extended to include a “center manifold” (corresponding to the Lyapunov space with exponent 0). 
Furthermore, (local) invariant manifolds can be defined not just for the stable and unstable subspace, 
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but for all Lyapunov spaces; see [BK94], [CK00], and [Rob98] for the necessary techniques and precise 
statements. 

Both the Grobman—Hartman theorem as well as the invariant manifold theorem can be extended to time 
varying systems, i.e., to linear skew product flows as described in Sections 56.5 to 56.8. The general situation 
is discussed in [BK94], the case of linearization at a periodic solution is covered in [Rob98], random 
dynamical systems are treated in [Arn98], and robust systems (control systems) are the topic of [CK00]. 


Definitions: 


A (nonlinear) differential equation in R‘ is of the form y = f(y), where f isa vector field on IR4@. Assume 
that f is at least of class C! and that for all yo € R?@ the solutions y(t, yo) of the initial value problem 
y(0; yo) = yo exist for allt € R. 

A point p € R¢ isa fixed point of the differential equation y = f (y) if y(t, p) = p forallt € R. 

The linearization of the equation y = f(y) at a fixed point p € R?¢ is given by x = Dy f (p)x, where 
Dy f (p) is the Jacobian (matrix of partial derivatives) of f at the point p. 

A fixed point p € R¢ of the differential equation y = f(y) is called hyperbolic if Dy f(p) has no 
eigenvalues on the imaginary axis, i.e., if the matrix Dy f (p) is hyperbolic. 

Consider a differential equation y = f(y) in R? with flow ® : R x R¢ —> R’, hyperbolic fixed point 
p and neighborhood U(p). In this situation the local stable manifold and the local unstable manifold 
are defined as 


W,,.(P) = {q € U: lim;oo ®(t,q) = p} and W;,.(p) = {q € U: lim;-.. P(t, q) = p}, 


respectively. 
The local stable (and unstable) manifolds can be extended to global invariant manifolds by following 
the trajectories, i.e., 


W*(p) = U,2oP(—6, W;,-(P)) and W"(p) = U9 P(t, Wis. (P))- 


Facts: 


Literature: [Arn98], [AP90], [BK94], [CK00], [Rob98]. 
See Facts 3 and 4 in Section 56.2 for dynamical systems induced by differential equations and their fixed 
points. 


1. (Hartman—Gobman) Consider a differential equation y = f(y) in R¢ with flow ®: R x R? —> 
R¢@. Assume that the equation has a hyperbolic fixed point p and denote the flow of the linearized 
equation x = Dy f(p)x by ¥ : R x R4 —+ R‘. Then there exist neighborhoods U(p) of p and 
V(0) of the origin in R’, anda homeomorphism h : U(p) —> V(0) such that the flows ® |u;p) 
and W |yq) are (locally) C°-conjugate, ie. h(®(t,y)) = W(t, h(y)) for ally ¢ U(p) andt € Ras 
long as the solutions stay within the respective neighborhoods. 

2. Consider two differential equations y = f;(y) in R? with flows ®; : R x R? —> R¢ fori = 
1,2. Assume that ©; has a hyperbolic fixed point p; and the flows are C*-conjugate for some 
k > 1 in neighborhoods of the p;. Then o (Dy fi(pi)) = o(Dy f2(p2)), ie., the eigenvalues of the 
linearizations agree; compare Facts 5 and 6 in Section 56.2 for the linear situation. 

3. Consider two differential equations y = f;(y) in R4 with flows ®; : R x R? —> R¢ fori = 1,2. 
Assume that ®; has a hyperbolic fixed point p; and the number of negative (or positive) Lyapunov 
exponents of Dy f;(p;) agrees. Then the flows ®; are locally C°-conjugate around the fixed points. 

4, (Invariant Manifold Theorem) Consider a differential equation y = f(y) in R? with flow ® : 
R x R? —> R’. Assume that the equation has a hyperbolic fixed point p and denote the linearized 
equation by x = Dy f (p)x. 


(i) There exists a neighborhood U(p) in which the flow ® has a local stable manifold W;,.(p) 
and a local unstable manifold W;.(p). 
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(ii) Denote by L~ (and L*) the stable (and unstable, respectively) subspace of Dy f (p); compare 
the definitions in Section 56.1. The dimensions of L~ (as a linear subspace of R“) and of 
W;,.(p) (as a topological manifold) agree, similarly for Lt and W;!(p). 


(iii) The stable manifold 
for Wi! .(p) and L*. 


loc 


(p) is tangent to the stable subspace L~ at the fixed point p, similarly 


$ 
loc 


5. Consider a differential equation y = f(y) in R@ with flow ® : R x R4 —> R*. Assume that the 
equation has a hyperbolic fixed point p. Then there exists a neighborhood U(p) on which ® is 
C°-equivalent to the flow of a linear differential equation of the type 


X; = —X;,X; € R%, 
&) = Xp X, € R*, 


where d, and d,, are the dimensions of the stable and the unstable subspace of Dy f (p), respectively, 
with d, + d, = d. 


Examples: 


1. Consider the nonlinear differential equation in R given by 2 + z — z* = 0, or in first-order form 


in R? 
a V2 = 
ale Peverlr 


The fixed points of this system are p; = [0, 0]", po = [1,0], ps = [—1,0]?. Computation of the 
linearization yields 


By 0 l 
1-143 0] 


Hence, the fixed point p, is not hyperbolic, while pz and p3 have this property. 
2. Consider the nonlinear differential equation in R given by Z + sin(z) + z = 0, or in first-order 


form in R2 
y y: 
i oe = fly). 
V2 —sin(y)) — y2 


The fixed points of the system are p,, = [7,0]? for n € Z. Computation of the linearization yields 


ee 0 1 
1 = —cos(y;) 1 ; 


Hence, for the fixed points p,, with n even the eigenvalues are 4), 2 = -} fami iV} with negative 
real part (or Lyapunov exponent), while at the fixed points p,, with n odd one obtains as eigenvalues 
Vj, = —; + V3 , resulting in one positive and one negative eigenvalue. Hence, the flow of the 


differential equation is locally C°-conjugate around all fixed points with even n, and around all 
fixed points with odd n, while the flows around, e.g., po and p, are not conjugate. 
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Given a dynamical system described by the ordinary differential equation (ODE) 
x(t) = f(t,x(t), u(t), x(t) = x°, 


where x is the state of the system and u serves as input, the major problem in control theory is to steer 
the state from x° to some desired state; i.e., for a given initial value x(f) = x° and target x!, can we find 
a piecewise continuous or L> (i.e., square-integrable, Lebesgue measurable) control function U such that 
there exists 4 > f% with x(t;;4) = x', where x(t; 4) is the solution trajectory of the ODE given above 
for u = 0? Often, the target is x! = 0, in particular if x describes the deviation from a nominal path. 
A weaker demand is to asymptotically stabilize the system, i-e., to find an admissible control function a 
(i.e., a piecewise continuous or L> function ti: [f, t;] H UYU) such that lim;. x(t; a) = 0). 

Another major problem in control theory arises from the fact that often, not all states are available for 
measurements or observations. Thus, we are faced with the question: Given partial information about the 
states, is it possible to reconstruct the solution trajectory from the measurements/observations? If this is the 
case, the states can be estimated by state observers. The classical approach leads to the Luenberger observer, 
but nowadays most frequently the famous Kalman—Bucy filter [KB61] is used as it can be considered as an 
optimal state observer in a least-squares sense and allows for stochastic uncertainties in the system. 

Analyzing the above questions concerning controllability, observability, etc. for general control systems 
is beyond the scope of linear algebra. Therefore, we will mostly focus on linear time-invariant (LTI) systems 
that can be analyzed with tools relying on linear algebra techniques. (For further reading, see, e.g., [Lev96], 
[Mut99], and [Son98].) 

Once the above questions are settled, it is interesting to ask how the desired control objectives can be 
achieved in an optimal way. The linear-quadratic regulator (LQR) problem is equivalent to a dynamic op- 
timization problem for linear differential equations. Its significance for control theory was fully discovered 
first by Kalman in 1960 [Kal60]. One of its main applications is to steer the solution of the underlying 
linear differential equation to a desired reference trajectory with minimal cost given full information on 
the states. If full information is not available, then the states can be estimated from the measurements 
or observations using a Kalman—Bucy filter. This leads to the linear-quadratic Gaussian (LQG) control 
problem. The latter problem and its solution were first described in the classical papers [Kal60] and [KB61] 
and are nowadays contained in any textbook on control theory. 
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In the past decades, the interest has shifted from optimal control to robust control. The question raised 
is whether a given control law is still able to achieve a desired performance in the presence of uncertain 
disturbances. In this sense, the LQR control law has some robustness, while the LQG design cannot be 
considered to be robust [Doy78]. The H.. control problem aims at minimizing the worst-case error that 
can occur if the system is perturbed by exogenous perturbations. It is, thus, one example of a robust control 
problem. We will only introduce the standard H,, control problem, though there exist many other robust 
control problems and several variations of the H,, control problem; see [GL95], [PUA00], [ZDG96]. 

Many of the above questions lead to methods that involve the solution of linear and nonlinear matrix 
equations, in particular Lyapunov, Sylvester, and Riccati equations. For instance, stability, controllability, 
and observability of LTI systems can be related to solutions of Lyapunov equations (see, e.g., [LT85, Sec. 13] 
and [HJ91]), while the LQR, LQG, and H,, control problems lead to the solution of algebraic Riccati 
equations, (see, e.g., [AKFIJ03], [Dat04], [LR95], [Meh91], and [Sim96]). Therefore, we will provide the 
most relevant properties of these matrix equations. 

The concepts and solution techniques contained in this chapter and many other control-related 
algorithms are implemented in the MATLAB® Control System Toolbox, the Subroutine Library in Control 
SLICOT [BMS*99], and many other computer-aided control systems design tools. 

Finally, we note that all concepts described in this section are related to continuous-time systems. 
Analogous concepts hold for discrete-time systems whose dynamics are described by difference equations; 
see, e.g., [Kuc91]. 


57.1 Basic Concepts 


Definitions: 


Given vector spaces 1 (the state space), U/ (the input space), and Y (the output space) and measurable 
functions f, g : [fo, tp] x & x Ut R", a control system is defined by 


x(t) = f(t, x(t), u(t)), 
y(t) = g(t, x(t), u(r), 
where the differential equation is called the state equation, the second equation is called the observer 


equation, and t € [f9, tr] (t7 € [0,00 ]). 
Here, 


X: [t,t] +> & is the state (vector), 
u: [t,t] +> U is the control (vector), 


y : [t, tf] +> Y is the output (vector). 
A control system is called autonomous (time-invariant) if 
f(t, x, u) = f(x, u) and g(t, x, u) = g(x, u). 


The number of state-space variables n is called the order or degree of the system. 

Let x! € R”. A control system with initial value x(f) = x° is controllable to x! in time t; > f if there 
exists an admissible control function u (i.e., a piecewise continuous or L» function u : [f, t;] H U) such 
that x(t); u) = x!. (Equivalently, (4), x!) is reachable from (tp, x°).) 

A control system with initial value x(t)) = x° is controllable to x! if there exists t, > f such that (t,x!) 
is reachable from (to, x°). 

If the control system is controllable to all x! € 2 for all (t%,x°) with x° € 4, it is (completely) 
controllable. 
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A control system is linear if ¥ = R",U = R”, Y = R?, and 


f(t, x, u) = A(t)x(t) + B(t)u(t), 
g(t, x, u) = C(t)x(t) + D(t)u(t), 


where A : [ttf] H R'*", B : [t,t] BH R'™",C : [f, tf] BH R?*", D : [t, tf] H R?*” are smooth 
functions. 
A linear time-invariant system (LTI system) has the form 


x(t) = Ax(t)+ Bu(t), 
y(t) = Cx(t)+ Du(t), 


with A € R’*", B ¢ R"*”, C € R’*", and D € R?*”, 
An LTI system is (asymptotically) stable if the corresponding linear homogeneous ODE x = Ax is 
(asymptotically) stable. (For a definition of (asymptotic) stability confer Chapter 55 and Chapter 56.) 
An LTI system is stabilizable (by state feedback) if there exists an admissible control in the form of a 
state feedback 


u(t) = Fx(t), F €R”*", 
such that the unique solution of the corresponding closed-loop ODE 
x(t) = (A+ BE)x(t) (57.1) 


is asymptotically stable. 
An LTI system is observable (reconstructible) if for two solution trajectories x(t) and x(t) of its state 
equation, it holds that 


Cx(t) = Cx(t) Vt < ty) (Vt > to) 


implies x(t) = X(t) Vt < t% (Vt > fp). 
An LTI system is detectable if for any solution x(t) of x = Ax with Cx(t) = 0 we have lim x(t) = 0. 
too 


Facts: 


1. For LTI systems, all controllability and reachability concepts are equivalent. Therefore, we only 
speak of controllability of LTI systems. 

2. Observability implies that one can obtain all necessary information about the LTI system from the 
output equation. 

3. Detectability weakens observability in the same sense as stabilizability weakens controllability: Not 
all of x can be observed, but the unobservable part is asymptotically stable. 

4. Observability (detectability) and controllability (stabilizability) are dual concepts in the following 
sense: an LTI system is observable (detectable) if and only if the dual system 


z(t) = A'z(t) + C' v(t) 
is controllable (stabilizable). This fact is sometimes called the duality principle of control theory. 


Examples: 


1. A fundamental problem in robotics is to control the position of a single-link rotational joint using 
a motor placed at the “pivot.” A simple mathematical model for this is the pendulum [Son98]. 
Applying a torque u as external force, this can serve as a means to control the motion of the 
pendulum (Figure 57.1). 
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FIGURE 57.1 Pendulum as mathematical model of a single-link rotational joint . 


If we neglect friction and assume that the mass is concentrated at the tip of the pendulum, 
Newton’s law for rotating objects 


mO(t) + mg sin @(t) = u(t) 


describes the counter clockwise movement of the angle between the vertical axis and the pendulum 
subject to the control u(t). This is a first example of a (nonlinear) control system if we set 


fe Pa 
x(t) = a |e > 
x(t) @(t) 

x2 


f(t, x, u) = } g(t,x,u) = x1, 


—mg sin(x;) 


where we assume that only ©(t) can be measured, but not the angular velocity O(t). 

For u(t) = 0, the stationary position O = x,© = 0 is an unstable equilibrium, ie., small 
perturbations will lead to unstable motion. The objective now is to apply a torque (control u) to 
correct for deviations from this unstable equilibrium, ie., to keep the pendulum in the upright 
position, (Figure 57.2). 

2. Scaling the variables such that m = 1 = g and assuming a small perturbation © — z in the inverted 
pendulum problem described above, we have 


sin® = —(@ — 2) + 0((@ — z)”). 


(Here, g(x) = o(x) if lim a) = 0.) This allows us to linearize the control system in order to 
x—-0O 


obtain a linear control system for g(t) := O(t) —z: 
g(t) — g(t) = u(t). 


This can be written as an LT system, assuming only positions can be observed, with 


<=|f]. a=[} ile a= |]. Cah Oh DH: 
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FIGURE 57.2 Inverted pendulum; apply control to move to upright position. 


Now the objective translates to: Given initial values x; (0) = (0), x2(0) = @(0), find u(t) to bring 
x(t) to zero “as fast as possible.” It is usually an additional goal to avoid overshoot and oscillating 
behavior as much as possible. 


57.2 Frequency-Domain Analysis 


So far, LTI systems are treated in state-space. In systems and control theory, it is often beneficial to use the 
frequency domain formalism obtained from applying the Laplace transformation to its state and observer 
equations. 


Definitions: 


The rational matrix function 
G(s) = C(sI— A) 'B+DeR?*"[s] 


is called the transfer function of the LTI system defined in section 57.1. 

In a frequency domain analysis, G(s) is evaluated for s = iw, where w € [0,00] has the physical 
interpretation of a frequency and the input is considered as a signal with frequency w. 

The L.o-norm ofa transfer function is the operator norm induced by the frequency domain analogue of 
the L»-norm that applies to Laplace transformed input functions u € L,(—00, 00; RR”), where L>(a, b; R”) 
is the Lebesgue space of square-integrable, measurable functions on the interval (a,b) C R with values 
inR”. 

The p x m-matrix-valued functions G for which ||G||,., is bounded form the space Loo. 

The subset of Lo, containing all p x m-matrix-valued functions that are analytical and bounded in the 
open right-half complex plane form the Hardy space Ho. 

The H,.-norm of G € Hy, is defined as 


IG |l a, = ess SUP Omax(G(i@)), (57.2) 
weR 
where Omax(M) is the maximum singular value of the matrix M and ess sup,¢,y h(t) is the essential 
supremum of a function h evaluated on the set M, which is the function’s supremum on M \ L where L 
is a set of Lebesgue measure zero. 


57-6 Handbook of Linear Algebra 


For T € R"*" nonsingular, the mapping implied by 
(A, B,C, D) + (TAT, TB, CT“, D) 


is called a state-space transformation. 

(A, B,C, D) is called a realization of an LTI system if its transfer function can be expressed as G(s) = 
Clsigs Ay BD: 

The minimum number # so that there exists no realization of a given LTI system with n < fis called 
the McMillan degree of the system. 

A realization with n = f is a minimal realization. 


Facts: 


1. If X, Y, U are the Laplace transforms of x, y, u, respectively, s is the Laplace variable, and x(0) = 0, 
the state and observer equation of an LTI system transform to 


sX(s) = AX(s) + BU(s), 
Y(s) = CX(s) + DU(s). 


Thus, the resulting input-output relation 


¥(s) = (C(sI — A)"'B + D) U(s) = G(s)U(s) (57.3) 


is completely determined by the transfer function of the LTI system. 

2. As a consequence of the maximum modulus theorem, H,, functions must be bounded on the 
imaginary axis so that the essential supremum in the definition of the H,.-norm simplifies to a 
supremum for rational functions G. 

3. The transfer function of an LTI system is invariant w.r.t. state-space transformations: 


D+ (CT )GI— TAT) (TB) = CsI, — A) B+ D=G(s). 
Consequently, there exist infinitely many realizations of an LT] system. 
4. Adding zero inputs/outputs does not change the transfer function, thus the order n of the system 


can be increased arbitrarily. 


Examples: 


1. The LTI system corresponding to the inverted pendulum has the transfer function 


=i 
-1 0 
G(s) = [1 9 E a + [0] = — 


2. The L.o-norm of the transfer function corresponding to the inverted pendulum is 


WGllz.. =1. 


3. The transfer function corresponding to the inverted pendulum is not in Ho. as G(s) has a pole at 
s = 1 and, thus, is not bounded in the right-half plane. 
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57.3 Analysis of LTI Systems 


In this section, we provide characterizations of the properties of LTI systems defined in the introduction. 
Controllability and the related concepts can be checked using several algebraic criteria. 


Definitions: 
A matrix A € R"*” is Hurwitz or (asymptotically) stable if all its eigenvalues have strictly negative real 


part. 
The controllability matrix corresponding to an LTI system is 


C(A, B) = [B, AB, A’B,..., A” |B] eR"™"™”. 


The observability matrix corresponding to an LTI system is 


C Atl 
The following transformations are state-space transformations: 


* Change of basis: 


xt Px for PeR"*" nonsingular, 
ut Qu for QeR”*” nonsingular, 
yr Ry for ReR?*? nonsingular. 


¢ Linear state feedback: urb> Fx+v,F € R”*",v: [to, tp] eR”. 
* Linear output feedback: ur> Gy+v,G € R”™? vy: [to, tp] > IR”. 


The Kalman decomposition of (A, B) is 


if Al Ag ale By nxn 
ViAV= so VBS » V €R”*" orthogonal, 
0 A; 0 


where (A), B,) is controllable. 
The observability Kalman decomposition of (A, C) is 


A; O 
, CW=[C, 0], W €R”*” orthogonal, 


where (A), C)) is observable. 


Facts: 
1. An LTI system is asymptotically stable if and only if A is Hurwitz. 
2. For a given LT] system, the following are equivalent. 
(a) The LTI system is controllable. 


(b) The controllability matrix corresponding to the LTI system has full (row) rank, Le., rankC(A, B) 
=n. 
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(c) (Hautus—Popov test) If p 4 0 and p* A = Ap*, then p*B 4 0. 
(d) rank([AI — A, B]) =n WA EC. 


The essential part of the proof of the above characterizations (which is “d)=-b)”) is an application 
of the Cayley-Hamilton theorem (Section 4.3). 
3. For a given LT] system, the following are equivalent: 


(a) The LTI system is stabilizable, ie. IF € R”*” such that A+ BF is Hurwitz. 
(b) (Hautus—Popov test) If p 4 0, p*A = Ap*, and Re(A) > 0, then p*B # 0. 
(c) rank([A —AI,B]) =n, VA € C with Re(A) = 0. 
(d) In the Kalman decomposition of (A, B), A3 is Hurwitz. 
4. Using the change of basis = Vx implied by the Kalman decomposition, we obtain 


xX) = A,X) + AyX> =p Byu, 


X> = A3X. 


Thus, X is not controllable. The eigenvalues of A; are, therefore, called uncontrollable modes. 
5. For a given LT] system, the following are equivalent: 


(a) The LTI system is observable. 


(b) The observability matrix corresponding to the LTI system has full (column) rank, i.e., rank O(A, C) = 
n. 


(c) (Hautus—Popov test), p 4 0, Ap = Ap => C'p £0. 
(d) rank ke e ‘| =n, WaeC., 


6. For a given LT] system, the following are equivalent: 


(a) The LTI system is detectable. 
(b) The dual system z = A'z + C!vis stabilizable. 
(c) (Hautus—Popov test) p 4 0, Ap = Ap, Re(A) > O=— C’pF 0. 


(d) rank ke B ‘| =n, Wd € Cwith Re(A) > 0. 
(e) In the observability Kalman decomposition of (A, C), A3 is Hurwitz. 


7. Using the change of basis = W‘x implied by the observability Kalman decomposition we obtain 


xX) = A,X + Byu, 
&% = Ak + A3k2 + Bou, 
= Cx. 


Thus, X is not observable. The eigenvalues of A; are, therefore, called unobservable modes. 

8. The characterizations of observability and detectability are proved using the duality principle and 
the characterizations of controllability and stabilizability. 

9. If an LTI system is controllable (observable, stabilizable, detectable), then the corresponding 
LTI system resulting from a state-space transformation is controllable (observable, stabilizable, 
detectable). 
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10. For A € R"*", B € R"*” there exist P € R"*", Q € R™*” orthogonal such that 


Ay Ais-1 As ny 


nN 


0 ai, 0 As—1,s—2 As—1,s—1 As—1,5 Ns—] 
or 0 0 Ase! te 


ny Ns_2 Ns] Ns 


nh m—-n, 


where 1, > 12 >... > Ns-1 > ns > 0, 05-1 > 0, Ajj_) = [Zjj_-) 0] € R"*"", Ujj;_) € R™*" 
nonsingular fori = 1,...,s — 1, U;_1,s~ is diagonal, and B, is nonsingular. 
Moreover, this transformation to staircase form can be computed by a finite sequence of 

singular value decompositions. 

11. An LTI system is controllable if in the staircase form of (A, B), n; = 0. 

12. An LTI system is observable ifn, = 0 in the staircase form of (A7, C7). 

13. An LTI system is stabilizable if in the staircase form of (A, B), As, is Hurwitz. 

14, An LTI system is detectable if in the staircase form of (A7,C T), A,, is Hurwitz. 

15. In case m = 1, the staircase form of (A, B) is given by 


Ai, «ee sans Ain by 
0 
a a“ 
paphe: |-" ,PB= 
0 


Ann—-1 Ann 


and is called the controllability Hessenberg form. The corresponding staircase from of (A’,C*) 
in case p = 1 is called the observability Hessenberg form. 


Examples: 


1. The LTI system corresponding to the inverted pendulum problem is not asymptotically stable as A 
is not Hurwitz: 0(A) = {+l}. 
2. The LTI system corresponding to the inverted pendulum problem is controllable as the controlla- 


bility matrix 
C(A, B) a 
Powe PINE Sf} 


has full rank. Thus, it is also stabilizable. 
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3. The LTI system corresponding to the inverted pendulum problem is observable as the observability 


matrix 
1 0 
O(A,C) = | 
0 1 


has full rank. Thus, it is also detectable. 


57.4 Matrix Equations 


A fundamental role in many tasks in control theory is played by matrix equations. We, therefore, review 
their most important properties. More details can be found in [AKFIJ03], [HJ91], [LR95], and [LT85]. 


Definitions: 


A linear matrix equation of the form 
AX+ XB=W, AeR™",BeER™",WeR™", 


is called Sylvester equation. 
A linear matrix equation of the form 


AX+XAT=W, AeR™',W= W!' €R™", 


is called Lyapunov equation. 
A quadratic matrix equation of the form 


0=Q+A'X+XA-— XGX, AeR™ G=G',Q=Q' e€R™", 


is called algebraic Riccati equation (ARE). 


Facts: 


1. The Sylvester equation is equivalent to the linear system of equations 
[Cm ® A) + (BT @ I,)| vec(X) = vec(W), 


where ® and vec denote the Kronecker product and the vec-operator defined in Section 10.4. Thus, 
the Sylvester equation has a unique solution if and only ifo(A) No(—B) = 9. 
2. The Lyapunov equation is equivalent to the linear system of equations 


[((Im ® A) + (A @ I,,)] vec(X) = vec(W). 


Thus, it has a unique solution if and only if 0(A) N o(—A’) = @. In particular, this holds if A is 
Hurwitz. 

3. If Gand Q are positive semidefinite with (A, G) stabilizable and (A, Q) detectable, then the ARE 
has a unique positive semidefinite solution X,, with the property that o(A — GX,,) is Hurwitz. 

4. Ifthe assumptions given above are not satisfied, there may or may not exist a stabilizing solution with 
the given properties. Besides, there may exist a continuum of solutions, a finite number of solutions, 
or no solution at all. The solution theory for AREs is a vast topic by itself; see the monographs 
[AKFIJ03], [LR95] and [Ben99], [Dat04], [Meh91], and [Sim96] for numerical algorithms to solve 
these equations. 
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Examples: 
1. For 


1/-3 3 
X=- 
Al De est 


Note that (A) = o(B) = {1,1} so thato(A)No(—B) = &. Thus, this Sylvester equation has the 
unique solution X given above. 


2. For 
0 1 0 0 1 0 
a | = | — | 
0 0 0 1 0 2 


the stabilizing solution of the associated ARE is 


is {—1, —l}. 
3. Consider the ARE 


0=C'C+A'X+XA—XBB'X 


corresponding to an LTI system with 


a= {74 le a= |i). C= [72> U4 DSO. 


-14+/3 0 


For this ARE, X = is a solution for all € € R. It is positive semidefinite for all 
0 

& > 0, but this ARE does not have a stabilizing solution as the LT] system is neither stabilizable nor 

detectable. 


57.5 State Estimation 


In this section, we present the two most famous approaches to state observation, that is, finding a function 
%(t) that approximates the state x(t) of a given LTI system if only its inputs u(t) and outputs y(t) are 
known. While the first approach (the Luenberger observer) assumes a deterministic system behavior, the 
Kalman-—Bucy filter allows for uncertainty in the system, modeled by white-noise, zero-mean stochastic 
processes. 
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Definitions: 


Given an LTI system with D = 0, a state observer is a function 
&: [0,00) BH R" 
such that for some nonsingular matrix Z € R"*” and e(t) = X(t) — Zx(t), we have 


lim e(t) = 0. 


t>oo 
Given an LTT system with stochastic disturbances 
x(t) = Ax(t)+ Bu(t)+ Bwi(s), 
y(t) = Cx(t) + v(t), 


where A, B,C are as before, B € R"*™, and w(t), v(t) are white-noise, zero-mean stochastic processes 
with corresponding covariance matrices W = W! € R”*” (positive semidefinite), V = V’ € R?*? 
(positive definite), the problem to minimize the mean square error 


E [|Ix(t) — X(¢)|I5] 


over all state observers is called the optimal estimation problem. (Here, E [r] is the expected value of r.) 


Facts: 


1. A state observer, called the Luenberger observer, is obtained as the solution of the dynamical 
system 


X(t) = H&(t) + Fy(t) + Gu(t), 


where H € R"*" and F € R"*? are chosen so that H is Hurwitz and the Sylvester observer 
equation 


AX —XA+FC=0 


has anonsingular solution X. Then G = X B and the matrix Z in the definition of the state observer 
equals the solution X of the Sylvester observer equation. 
2. Assuming that 


* wand vare uncorrelated stochastic processes, 
* the initial state x° is a Gaussian zero-mean random variable, uncorrelated with w and v, 


¢ (A, B) is controllable and (A, C) is observable, 


the solution to the optimal estimation problem is given by the Kalman—Bucy filter, defined as the 
solution of the linear differential equation 


&(t) = (A— Y,C7V7'C)&(t) + Bu(t) + Y,C7V y(t), 
where Y, is the unique stabilizing solution of the filter ARE: 
0= BWB’ + AY+YA™—YCTV'CY. 


3. Under the same assumptions as above, the stabilizing solution of the filter ARE can be shown to be 
symmetric positive definite. 
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Examples: 


1. A Luenberger observer for the LTI system corresponding to the inverted pendulum problem can be 
constructed as follows: Choose H = diag(—2,—}) and F = [2 1] ” ‘Then the Sylvester observer 
equation has the unique solution 


Note that X is nonsingular. Thus, we get G = XB = 5 a 2 4]. 


2. Consider the inverted pendulum with disturbances v, w and B= [1 1] om Assume that V = W= 
1. The Kalman—Bucy filter is determined via the filter ARE, yielding 


= (14+ 72) F iE 
ie | 


Thus, the state estimation obtained from the Kalman filter is given by the solution of 


; -1-/2 1]. 0 1 
k(n) = 5 i X(t) + " u(t) + (1+ V2) ] y(t) 


57.6 Control Design for LTI Systems 


This section provides the background for some of the most important control design methods. 


Definitions: 


A (feedback) controller for an LTI system is given by another LT] system 


rt) = Er(t)+ Fy(t), 
u(t) = Hr(t)+ Ky(t), 


where E € RN*, F € RN*?, H Ee R™N, K € R™”?, and the “output” u(t) of the controller serves as 
the input for the original LTI system. 

If E, F, H are zero matrices, a controller is called static feedback, otherwise it is called a dynamic 
compensator. 

A static feedback control law is a state feedback if in the controller equations, the output function y(t) 
is replaced by the state x(t), otherwise it is called output feedback. 

The closed-loop system resulting from inserting the control law u(t) obtained from a dynamic compen- 
sator into the LTI system is illustrated by the block diagram in Figure 57.3, where w is as in the definition 
of LTI systems with stochastic disturbances in Section 57.5 and z will only be needed later when defining 
the H,. control problem. 

The linear-quadratic optimization (optimal control) problem 


min J(u), where J(u 


uel 3(0,00;U) 


=3] 0" )? Qy(t) + u(t) " Ru(t)) dt 


aii 
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r=Er+Fy 
u=Hr+Ky 


FIGURE 57.3 Closed-loop diagram of an LTI system and a dynamic compensator. 


subject to the dynamical constraint given by an LTI system is called the linear-quadratic regulator (LQR) 
problem. 
The linear-quadratic optimization (optimal control) problem 


ty 


min J(u), where /(u) = lim eae: J otaye + un" Rutn) dt 


ucL>(0,00;U/) tp—oo 2tf 
ty 


subject to the dynamical constraint given by an LTI system with stochastic disturbances is called the 
linear-quadratic Gaussian (LQG) problem. 

Consider an LTI system where inputs and outputs are split into two parts, so that instead of Bu(t) we 
have 


By, w(t) + Bult), 
and instead of y(t) = Cx(t) + Du(t), we write 


z(t) = C,x(t)+ Dyw(t) + Dyu(t), 
y(t) = Cx(t) + Dy, w(t) + Dyu(t), 


where u(t) € R” denotes the control input, w(t) € R™ is an exogenous input that may include noise, 


linearization errors, and unmodeled dynamics, y(t) € R” contains measured outputs, while z(t) ¢ R” 


Gu Gy 


Cn =| denote the corresponding transfer 


is the regulated output or an estimation error. Let G = 


Z)} {Gu Gn 
Y Gr, Ga 

where Y, Z, U, W denote the Laplace transforms of y, z, u, w. 
The optimal H,, control problem is then to determine a dynamic compensator 


function such that 


W 
U 


> 


rt) = Er(t)+ Fy(t), 
Hr(t) + Ky(t), 


u(t) 
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with E ¢ RN*‘,F e RN*e, He R™*N K € R™*? and transfer function M(s) = H(sI—E)~!F+K 
such that the resulting closed-loop system 

X(t) = (A+ BK Z,Cz)x(t) + (By Z2H)r(t) + (Bi + Bo K Z; Dy )w(t), 

f(t) = FZ,Cox(t) + (E + FZ, DyH)r(t) + FZ, Daw(t), 

2(t) = (Ci + DipZ.K Cy)x(t) + DinZ2.Hr(t) + (Dur + DirK Z; Dx) w(t), 
with Z; = (I — Dy K)~' and Z, = (I — KD)", 


* is internally stable, i.c., the solution of the system with w(t) = 0 is asymptotically stable, and 
* the closed-loop transfer function T;y (s) = G22(s)+ Ga1(s)M(s)(I — Gy, (s)M(s))+G y2(s) from 
w to z is minimized in the H,.-norm. 


The suboptimal H,, control problem is to find an internally stabilizing controller so that 
I ey I Axo <Y> 


where y > 0 is a robustness threshold. 


Facts: 


1. If D = Oand the LTI system is both stabilizable and detectable, the weighting matrix Q is positive 
semidefinite, and R is positive definite, then the solution of the LQR problem is given by the state 
feedback controller 


u,(t) = — R7'B’X,x(t), t>0, 
where X,, is the unique stabilizing solution of the LQR ARE, 
0=C™QC+A'X+XA— XBR 'B'X, 


2. The LQR problem does not require an observer equation — inserting y(t) = Cx(t) into the cost 
functional, we obtain a problem formulation depending only on states and inputs: 


lo.@) 


Tw) = 5 i (y(t) Qy(t) + u(t)” Ru(e)) dt 
0 


lo. @) 


= 5 | GuTcT actt) + u(r" Ru(n) dt. 
0 


3. Under the given assumptions, it can also be shown that X,, is symmetric and the unique positive 
semidefinite matrix among all solutions of the LQR ARE. 

4, The assumptions for the feedback solution of the LQR problem can be weakened in several aspects; 
see, e.g., [Gee89] and [SSC95]. 

5. Assuming that 


* wand vare uncorrelated stochastic processes, 

* the initial state x° is a Gaussian zero-mean random variable, uncorrelated with w and v, 
¢ (A, B) is controllable and (A, C) is observable, 

the solution to the LQG problem is given by the feedback controller 


u(t) = —R7!B' X,x(t), 
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where X,, is the solution of the LQR ARE and x is the Kalman—Bucy filter 
&(t) = (A— BR'BTX, — Y,C7V~'c)a(t) + YC V-yQ2), 


corresponding to the closed-loop system resulting from the LQR solution with Y,. being the stabi- 
lizing solution of the corresponding filter ARE. 

6. In principle, there is no restriction on the degree N of the Ho. controller, although smaller dimen- 
sions N are preferred for practical implementation and computation. 

7. The state-space solution to the H,. suboptimal control problem [DGKF89] relates H. control to 
AREs: under the assumptions that 


¢ (A, Bx) is stabilizable and (A, C;) is detectable for k = 1,2, 
. Dy = 0, Dy = 0, and 


B 0 
Pile Pnj=to a}. |p| os= |) 


a suboptimal H,, controller exists if and only if the AREs 


1 
0= CPC, + AX+ XAT +X (m7 — BB) x, 
y 
1 
o= Bray + aT +va+y (Sorct - cc!) ¥ 
Y 


both have positive semidefinite stabilizing solutions X. and Y,, respectively, satisfying the spectral 
radius condition 


p(xY) < y?. 


8. The solution of the optimal H,, control problem can be obtained by a bisection method (or any 
other root-finding method) minimizing y based on the characterization of an H.. suboptimal 
controller given in Fact 7, starting from yp for which no suboptimal H,, controller exists and y; 
for which the above conditions are satisfied. 

9. The assumptions made for the state-space solution of the H.. control problem can mostly be 
relaxed. 

10. The robust numerical solution of the H,. control problem is a topic of ongoing research — the 
solution via AREs may suffer from several difficulties in the presence of roundoff errors and should 
be avoided if possible. One way out is a reformulation of the problem using structured generalized 
eigenvalue problems; see [BBMX99b], [CS92] and [GL97]. 

11. Once a (sub-)optimal y is found, it remains to determine a realization of the H, controller. One 
possibility is the central (minimum entropy) controller [ZDG96]: 


= na aE 2 sh _ iE 
E=A+ —BiBi = ByBy Xoo ZeoVooC Co, 
y 
PeZgiaG. K&B xe =, 


where 


Control Theory 57-17 


Examples: 


1. The cost functional in the LQR and LQG problems values the energy needed to reach the desired 
state by the weighting matrix R on the inputs. Thus, usually 


R= diag(p1,. a > Pm): 


The weighting on the states or outputs in the LQR or LQG problems is usually used to penalize 
deviations from the desired state of the system and is often also given in diagonal form. Common 
examples of weighting matrices are R = pI, Q = y I, for p,y > 0. 

2. The solution to the LQR problem for the inverted pendulum with Q = R = 1 is given via the 
stabilizing solution of the LQR ARE, which is 


P Paes 2 | 
14+ 72 Sri eg2\ 


resulting in the state feedback law 


u(t) = — 14 vi ViVi + V3] x(0). 


The eigenvalues of the closed-loop system are (up to four digits) o(A— BR! B?X,) = {-1.09874+ 
0.4551i}. 

3. The solution to the LQG problem for the inverted pendulum with Q, R as above and uncertainties 
v,wwith B = [1 1] " is obtained by combining the LQR solution derived above with the Kalman-— 
Bucy filter obtained as in the examples part of the previous section. 

Thus, we get the LQG control law 


u(t) = —[1+ v2 v3v14 v2 2 | at), 


where x is the solution of 


lene = 
(t 


X(t) = — 


ee wk (t) + ( iva] 
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58.1 Introduction 


Fourier analysis has been employed with great success in a wide range of applications. The underly- 
ing theory is based on a small set of linear transforms on particular linear spaces. It may, in fact, 
be best to refer to two parallel theories of Fourier analysis—the function/functional theory and the 
discrete theory—according to whether these linear spaces are infinite or finite dimensional. The func- 
tion/functional theory involves infinite dimensional spaces of functions on R” and can be further divided 
into two “subtheories’—one concerned with functions that are periodic on R” (or can be treated as 
periodic extensions of functions on finite subregions) and one concerned with more general functions 
and functionals on R”. This theory played the major role in applications up to half a century or so ago. 
However, the tools from the discrete theory (involving vectors in C% instead of functions on R") are 
much more easily implemented on digital computers. This became of interest both because much of the 
function/functional theory analysis can be closely approximated within the discrete theory and because 
the discrete theory is a natural setting for functions known only by samplings. In addition, “fast” algo- 
rithms for computing the discrete transforms were developed, allowing discrete analysis to be done very 
quickly even with very large data sets. For these reasons the discrete theory has become extremely im- 
portant in modern applications, and its utility, in turn, has greatly extended the applicability of Fourier 
analysis. 

In the first part of this chapter, elements of the function/functional theory are presented and illus- 
trated. For expediency, attention will be restricted to functions and functionals over subsets of R.A 
corresponding review of the analogous elements of the discrete theory then follows, with a discussion of 
the relations between the two theories following that. Finally, one of the fast algorithms is described and 
the extent to which this algorithm improves speed and applicability is briefly discussed. 

The development here is necessarily abbreviated and covers only a small fraction of the theory and 
applications of Fourier analysis. The reader interested in more complete treatments of the subject is 
encouraged to consult the references given throughout this chapter. 
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58.2 The Function/Functional Theory 


Definitions: 


The following function spaces often arise in Fourier analysis. In each case Z is a subinterval of R, and all 
functions on Z are assumed to be complex valued. 


* C°(Z), the normed linear space of bounded and continuous functions on Z with the norm || f|| = 
sup{| f(x)|: x € Z}. 

* £'(Z), the normed linear space of absolutely integrable functions on Z with the norm || f|| = 
Jel f@)| dx. 


* £7(Z), the inner product space of square integrable functions on Z with the inner product ( f,g) = 


Le f (x) g(x) dx. 


If Z is not specified, then Z = R. 
Let @ € £!(R). The Fourier transform F[@] and the inverse Fourier transform F~'[@] of @ are the 
functions 


Fléllx ai b(yje?"™'dy and Fell. = b(yje?"'dy. 


Additionally, the terms “Fourier transform” and “inverse Fourier transform” can refer to the processes for 
computing these functions from ¢. 
A function ¢ on R is said to be periodic if there is a positive value p, called a period for ¢, such that 


o(x+ p)=G(x) VeeER. 


The smallest period, if it exists, is called the fundamental period. 
The Fourier series for a suitably integrable, periodic function ¢ with period p > 0 is the infinite series 


CO 
S- Ck ei2tonx 
k=—00 
where 
k 1 [? : 
O=— and Ch= ~ | (ye 27" dy. 
P P Jo 


The theory for Fourier transforms and series can be generalized so that the requirement of ¢ being 
“suitably integrable” can be greatly relaxed (see [How01, Chap. 20, 30-34] or [Str94, Chap. 1-4]). Within 
this generalized theory the delta function at a € R, 5,, is the functional limit 


S.(x) = li 1 et ‘ ee 1 if|s|<e 
a(x) = lim —pulse,(x — a where ulse.(s) = ; ; 
€>0+ 2? 4 0 if|s| >e 


That is, 5, is the (generalized) function such that 


i: W(x)bq(x) dx = tim. | Ue Sa aide 
rare e>0* Jig 2€ 


whenever 7 is a function continuous at a. 
An array of delta functions is any expression of the form 


foe) 


SS CEOkAx 


k=-—0o 
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where Ax > 0 is fixed (the spacing of the array) and c; € C for each k € Z. If the array is also periodic 
with period p, then the corresponding index period is the positive integer N such that 


p= NAx and grin = oe Vk EZ. 


The convolution ¢ * wy of a suitably integrable pair of functions ¢ and yy on R is the function given by 


bx v(x) =i =O D: 


Facts: 
All the following facts except those with specific reference can be found in [How01] or [Str94]. 


1. Warning: Slight variations of the above integral formulas, e.g., 


1 0 : ] a ; 
a Ry =a ixy 
= | eee sane = | sera 


are also often used to define, respectively, F[@] and F—![@] (or even F—![@] and F[@]). There is 
little difference in the resulting “Fourier theories,” but the formulas resulting from using different 
versions of the Fourier transforms do differ in details. Thus, when computing Fourier transforms 
using different tables or software, it is important to take into account the specific integral formulas 
on which the table or software is based. 

2. Both F and F~! are continuous, linear mappings from L1(R) into C°(R). Moreover, if both ¢ and 
F [@] are absolutely integrable, then F~'[F[]] = ¢. 

3. In the generalized theory (which will be assumed hereafter), F and F—' are defined ona linear space 
of (generalized) functions that contains all elements of £!(R) and C?(R), all Fourier transforms 
of elements of £'(IR) and £?(R), all piecewise continuous periodic functions, the delta functions, 
and all periodic arrays of delta functions. F and F~! are one-to-one linear mappings from this 
space onto this space, and are inverses of each other. 

4, Every nonconstant, piecewise continuous periodic function has a fundamental period, and every 
period of such a function is an integral multiple of that fundamental period. Moreover, any two 
Fourier series for a single periodic function computed using two different periods will be identical 
after simplification (e.g., after eliminating zero-valued terms). 

5. The Fourier series )>,.7 cxe’?"** for a periodic function ¢ with period p can be written in 


trigonometric form 
CO 
cot » [ax cos(27 wx) + by sin(27 w,x)] 
k=l 
where 
2 P 
ap =Ch+C_p~= -[ (x) cos(27 a,x) dx 
P Jo 
and 


2 P 
by = icy — ic_~ = =f (x) sin(27 wx) dx . 
0 


6. OnZ = (0, p) (or any other interval Z of length p), the exponentials 


1 eenPr :O=k/prke Z} 
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10. 


11. 


12. 
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form an orthogonal set in £7(Z), and the Fourier series > kez Ck ex for a periodic function @ 
with period p is simply the expansion of ¢ with respect to this orthogonal set. That is, 


(d(x), ef 2x) 


|jei27 ox |] 2 Vk eZ. 


C= 
Similar comments apply regarding the trigonometric form for the Fourier series and the set 
{1, cos(27@zx), sin(27 a,x) : wp =k/p,k EN}. 
A periodic function ¢ can be identified with its Fourier series )>,-7 cxe'*"** under a wide range 
of criteria. In particular: 


* If @ is smooth, then its Fourier series converges uniformly to @. 


* If @ is piecewise smooth, then its Fourier series converges pointwise to @ everywhere @ is con- 
tinuous. 


* If @ is square-integrable on (0, p), then 


P N 2 
suv! cay f ue = D_o ei 2Toux dx = 0. 
* Within the generalized theory, 
lees [. ee) low a Do all dx =0 


whenever 7 is a sufficiently smooth function vanishing sufficiently rapidly at infinity (e.g., any 
Gaussian function). 


. [BH95, p. 186] Suppose ¢ is a periodic function with Fourier series )>, 27 c.e'?7". If p is m-times 


differentiable on R and @” is piecewise continuous for some m € N, then there is a finite constant 
B such that 


Ick] Vk eZ. 


Ik" 


. For each a € R and function ¢ continuous at a: 


/ P(x)da(x)dx= (a) and = pd, = H(a)da. 


For eacha € R: 
6 Fldallx es e i2max and Fo! [e274 | = Sq. 
A F [e?2™*] |, = 5a and Fo [8a] lx = ei2max 


A function ¢ is periodic with period p and Fourier series }°,-7 ¢ pel?" kx ifand only if its transforms 
are the arrays 


lo.@) lo,@) 


Fi@l= So cedran and = FM) = S04 8kao 


k=—00 k=—00 


with spacing Aw = 1/p. 
If 


g= S- rokAx 


k=—o0o 


is a periodic array with spacing Ax, period p, and corresponding index period N, then: 


Fourier Analysis 58-5 


* The Fourier series for ¢ is given by 


oe) 
p ® ei2TOnx 
n 


where w, = n/p = n/(NAx) and 


l N-1 


®,, = e i2akn/N 
NAx Da 


¢ The Fourier transforms of ¢ are also periodic arrays of delta functions with index period N. Both 
have spacing Aw = 1/p and period P = 1/Ax. These transforms are given by 


Fl] = yo DSnaw and F'g] = S- P_1dnaw- 


n=—OoO n=—OoOoO 


13. (Convolution identities) Let f, F, g, and G be functions with F = F[f] and G = F[g]. Then, 
provided the convolutions exist, 


Flfg]=FxG and Flf*xgl=FG. 


Examples: 


1. Fora > 0, 


Fipulses= f pulse, (y)e~?"*” dy 


lo) 
a 
= —idaxy J 
= e y 
—a 
el2max _ ei2max sin(27 ax) 
i2nx ux 


2. Let f be the periodic function, 


oe |x| for -l<x<l 
IIE Ge 5 pee aneR. 


The period of this function is p = 2, and its Fourier series }>,-7 cre?" is given by 
k 4 oy: 
Wk =_- _ (so e! TT OKX — em *) 
2 


and 


ae |x| ifx<1|[ ay 
Ch=- I e dx 
2Jo ||x-—2| ifl<x 


_ 1/2 ifk =0 
| (kr)? [(-)'-1) ifk 40° 
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Since f is continuous and piecewise smooth, f equals its Fourier series, 


mb ee 
f=5+ SS (a) is VxeR. 


ken 
keZ\{0} 
Moreover, 
ie pe 
= 5°04 oo) k/2 « 
keZ\{0} 
3. Let 


g= S- PO ax 


k=—00 


be the periodic array with spacing Ax = 1/3, index period N = 4, and coefficients 


og =0, @&=1, @&=2, and g=1. 


The period p of ¢ is then 


NA 4 pase 
= x= . = = 
P 3 3 


Its Fourier transform ® = F[@] is also a periodic array, 


lo.e) 
P= Yo Odnaw, 


n=—Oo 


with index period N = 4. The spacing Aw and period P of ® are determined, respectively, from 
the period p and spacing Ax of ¢ by 


1 1 3 1 1 
Aw = — = —- = - and P= — 
p 48 4 


The coefficients are given by 


N-1 


1 : 
©, _ e i2akn/N 
NAx do 


k=0 


fos 1 —i2n0n/4 —i2m1n/4 —i2m2n/4 —i2m3n/4 
= ae [doe + die + gre + p3e | 


= 0e° a le win /2 4 eine as le i3n7/2 
4(1/3) 


=F [0+ (i+ 24H", 


Thus, 


Dy = 3 ee d,=0 nd eee 
= 3, =-, =0 a =x. 
0 5 2 = 5 
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Applications: 


1. [BCO1, p. 155] or [Col88, pp. 159-160] (Partial differential equations) Using polar coordinates, the 
steady-state temperature u at position (r,@) on a uniform, insulated disk of radius 1 satisfies 
5 07u du 0? 


P apaa Pt aa. appl forO <r <1. 


As a function of @, u is periodic with period 27 and has (equivalent) Fourier series 


foe) 


S- cpe* and cot Ss [ax cos(kx) + by sin(kx)] 


k=—00 k=1 


where the coefficients are functions of r. If u satisfies the boundary condition 
u(1,0) = f(@) for0 <6 < 27 


for some function f on [0, 277), then the coefficients in the series can be determined, yielding 


u(r,0) = [3 rk ypetk? =yt ye [rex cos(k@) + r* By sin(k0)] 
k=—oo k=1 


where 


1 20 . 
= — 6)e'* do, 
Vk =f f (Ae 


2n 2a 
a, = - | f (9) cos(k@) dé, and Be = - | f (0) sin(k@) dé . 
H Jo T Jo 


2. (Systems analysis) In systems analysis, a “system” S transforms any “input” function f; to a corre- 
sponding “output” function fo = S[ f;]. Often, the output of S can be described by the convolution 
formula 


fo =h* fi 


where h is some fixed function called the impulse response of the system. The corresponding Fourier 
transform H = F[h] is the system’s transfer function. By the convolution identity, the output is 
also given by 


fo =F [HF] where F; = F[ fr] . 
Two such systems are 
* A delayed output system, for which 
h(x) =6r(x) and  H(w) =e "7° 


for some T > 0. Then 
fo(x) = fr * dr(x) = / filx — y)br(y)dy = fr(x — T). 


* [ZTF98, p. 178] or [How01, p. 477] An ideal low pass filter, for which 


h(x) = snes) and H(y) = pulseg(y) 
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for some Q > 0. If an input function f; is periodic with Fourier series }>,.7 ce!" **, then 


CO 
HF; = pulseg - S- CKOey 
k=—00 
CO 

= S- Cx pulseg (We) de, 

k-oo 
- Saf) if leo < ahha 2 3 So. 

ae 0 ifjax| > Q aes 


Thus, 


— S Ck ei2Tonx 


x lox |S 


post Arie? * va] 


|o|S2 
3. [ZTF98, pp. 520-521] (Deconvolution) Suppose S is a system given by 
S[ fr] =h» fr, 
but with h and H = F[h] being unknown. Since 
fea Fo LHF: where F; = F[ fr], 


both h and H can be determined as follows: 
* Find the output fo = S| fr] for some known input f; for which F; = F[ fr] is never zero. 


* Compute 


H= 
Fy; 


where Fo = F[ fo]. 


* Compute h = F~![H]. 


Similarly, an input f; can be reconstructed from an output fo by 
| FE 
fr=F" =| 


provided the transfer function H is known and is never zero. 


58.3 The Discrete Theory 


Definitions: 


In all the following, N € N, and the indexing of any “N items” (including rows and columns of matrices) 
will run from 0 to N — 1. 
An Nth order sequence is an ordered list of N complex numbers, 


(co; C1, C2, ++ oy Cn-1). 
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Such a sequence will often be written as the column vector 
T 
c= [co, C1, C2) --65 cn-1] > 


and the kth component of the sequence will be denoted by either cx or [c],. In addition, any such sequence 
will be viewed as part of an infinite repeating sequence 


G ++ €-1) CQ C1, C2) ---5 CN-1) CN) -- ) 
in which cyi,4 = cx for allk € Z. 
Let c be an Nth order sequence. The (Nth order) discrete Fourier transform (DFT) Fyne and the (Nth 


order) inverse discrete Fourier transform (inverse DFT) 7. wee of c are the two Nth order sequences 
given by 


= 1 me 
[Fnue]n = —= > Ck ei2tkn/N and [ ele = S- Cn ei2mnk/N 


Additionally, the terms “discrete Fourier transform” and “inverse discrete Fourier transform” can refer to 
the processes for computing these sequences from c. 

Given two Nth order sequences a and b, the corresponding product ab and convolution a « b are the 
Nth order sequences given by 


1 
[ab], =axb, and = [axb], = i S- ay_;b;. 


Facts: 
All the following facts can be found in [BH95, Chap. 2, 3] or [How01, Chap. 38, 39]. 


1. Warning: Slight variations of the above summation formulas, e.g., 


N-1 l N-1 

S Ck e i2akn/N and a Cn fet nk/N | 
N 

k=0 n=0 


are also often used to define, respectively, F we and F ae (or even F ye and F NC). 

2. Fy and F N_ are one-to-one linear transformations from C% onto CN. They preserve the standard 
inner product and are inverses of each other. 

3. Letting w = e~!2"/N, the matrices (with respect to the standard basis for CN) for the Nth order 
discrete Fourier transforms (also denoted by F ‘~ and F are 


1 1 1 1 1 

i apae wh wh3 wi(N-1) 
. ei wea yw we w2(N-1) 
Fy = YN 1 wel w22 w3 w3(N-1) 


1 wiN=DI w(N=D2 y(N=D3- yy (N=D(N=1) 
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and 
1 1 1 1 1 
l wohl wl wi3 w7lN=) 
l wo2l w-22 w—23 w2N=) 
Fe = = l wl w-32 w733 w(N-D 
Lo wrQW-Di wrN-D2 w-(N=-3 0. yo (N-DIN-1) 


4. Fuand Ff, is are symmetric matrices, and are complex conjugates of each other (i.e., LF N] “fF rg ). 


5. (Convolution identities) Let v, V, w, and W be Nth order sequences with V = a wv and W = - NW. 
Then 


Fylvw] = V*W and Fylv * w] = VW. 


Examples: 
1. The matrices for the four lowest order DFTs are F 1= [1], 


2 2 2 
zm 1 {1 1 pe 1 
A= F/I |: Te Je 2 -1-V3 -14V31, 
; -1+ 73 1 | 
and 
1 1 1 1 
aS 1 ]1l -—-1 1 1 
Fxz=- 
2/1 -1 1 -1 
1 1 —-l -1 


2. The DFT of v = [1, 2, 3, 4]" is V = [Vo, Vi, Vo, V3]? where 


Vo Ka ae efi 5 
Vi 1/1 -i 1 i 2 2+1 
1 eee a ees a ee 
V; ps a Sella a ey 


Applications: 
1. [CG00, p. 153] Assume 


2M 
faa ie 
n=0 


is the product of two known Mth order polynomials 


M 
g(z)= Se gaz and h(z) = > hyzk : 
k=0 
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The coefficients of f form a sequence f = [fo, fis fas... fom)" of order N = 2M + 1. 
These coefficients can be computed from the coefficients of g and h using either of the following 
approaches: 


° Let g = [go, 81, Br -- +> gom)? and h = [ho, hy, ho, ..., hoy)” be the Nth order sequences in 
which g; and h, are the corresponding coefficients of the polynomials g and h when k < M and 
are zero when M < k < N. The coefficients of f can then be computed by 


2M 
fe=Y— gx-jhj =VNigeh], fork =0,1,2,..., 2M. 
j=0 


* Let F = [Fo, Fi, Fo, ..., Fou]? be the Nth order DFT of f, and note that, for n = 0, 1, 2,..., 
N-1, 


1 N-1 sped 1 1 
a eed —i2akn/N n n n 
tag fee og te oi )g(w") 


where w = e~'?”/‘. Thus, the coefficients of f can be computed by first using the formulas for 
g and h to compute 

1 
VN 


and then taking the inverse DFT of F = [Fo, F), Fo, ...5 Fy-i]’. 


k= h(w")g(w") forn=0,1,2,..., N—-1, 


2. [BH95, p.247] Consider finding the solution v = [vo, V1, V2, ..-5 vy_i]_ tothe difference equation 
avVp-1 + PvE + Ovesi = fe forn=0,1,2,..., N-1 


where @ and 8 are constants and f = [ fo, fi, fos...» fw—i]” isa known Nth order sequence. If 
the boundary conditions are periodic (ie., vp = vy and v_; = vy_y), then setting 


v=Fi'V and f=F.'F 
yields the sequence v that satisfies the periodic boundary conditions, and whose DFT, V, satisfies 


N-1 
C, ei2tnk/N — 9 fork =0,1,2,...,N—1 


where 


Cres BEV 3g ees 
a cos {| —— na Fa. 
. N 


From this, it follows that the solution v to the original difference equation is the DFT of the sequence 
[Vo Vi, Va, me) Vy-i]" given by 


provided the denominator never vanishes. 

3. A difference equation of the form just considered with periodic boundary conditions arises when 
considering the steady-state temperature distribution on a uniform ring containing heat sources 
and sinks. In this case, the temperature v, as a function of angular position 0, is modeled by the 
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one-dimensional Poisson’s equation 
d’v 
qi es 


where f (@) describes the heat source/sink density at angular position @. The discrete analog of this 
equation is 


Ve-1 — 2Ve + Veg = fe forn=0,1,2,..., N-1 
where 
ie and f= [ fos fis fas sey fui]? 


describe, respectively, the temperatures at N evenly space positions around the ring, and the net 


VS [vo, Vi> V2> +++) VN-1 


sources and sinks of thermal energy about these positions. Setting 
v=Fi'V and f=F.'F 
and applying the formulas given above yields 


2m0n 
[2cos (F") — 2 Vi. F; forn = 0,1, 2,..., N—-1. 


The coefficient on the left is nonzero when n ¥ 0. Thus, 


F 
Vp = ——4.—_— forn =1,2,...,N—1. 
2cos (4) —2 


However, for n = 0, 


, Xa , wa 
0-Y= fy = — fe ORROIN — __ feo 
Tad, Tae 
pointing out that, for an equilibrium temperature distribution to exist, the net applied heat energy, 


pees fk, must be zero. Assuming this, the last equation then implies that Vo is arbitrary, which, 
since 


; oS jo NS 
Y= a: e Atk O/N Son 
VN VN 


means that the given conditions are not sufficient to determine the average temperature throughout 
the ring. 


58.4 Relating the Functional and Discrete Theories 


Definitions: 


Let f be a (continuous) function on a finite interval [0, L]. For any N € N and Ax > 0 satisfying 
NAx < L, the corresponding (Nth order) sampling (with spacing Ax) is the sequence 


f= [ots biisgetaal where f, = f(kAx). 


Corresponding to this is the scaled sampling 


f= fos Fis fs rr) Peale 
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and the discrete approximation 


where, in both, 


zt frAx fork =0,1,2,...,N-—1 
feHh x } . 
fk+n in general 


Facts: 


1. [How01, pp. 713-715] Let f be the discrete approximation of a continuous function f based on 
an Nth order sampling with spacing Ax. Then f is a periodic array of delta functions with spacing 
Ax and index period N that approximates f over the interval (0, NAx). In particular, for any 
other function y which is continuous on [0, NAx], 


NAx NAx be ~~ 
/ w(x) f(x) dx © | w(x) f(x) dx = y W(kAx) fx - 
0 0 


Attempting to approximate f with f outside the interval (0, NAx), however, cannot be justified 
unless f is also periodic with period NAx. 
2. [How01, chap. 38] Let 


f= [ fos fis fay ves fral? and F = [F), Fi, Fy, ...5 Fi)" 
be two Nth order sequences, and let 


CO [oe] 
f= S- fie Sk ax and F= S- Frdnaw 


k=—00 n=—0o 


be two corresponding periodic arrays of delta functions with index period N, and with spacings 
Ax and Aw satisfying Ax Aw = 1/N. Then 


F=F[f] — > F=AoJ/NAyf. 
In particular, if Aw = Ax = N~"?, then 
F=F[f] — > F=Afyf. 


3. [How01, pp. 719-723] Suppose f is a continuous, piecewise smooth function that vanishes outside 
the finite interval (0, L). Let F = F[f], and let f be the Nth order scaled sampling of f with 
spacing Ax chosen so that L = (N — 1)Ax. Then, for each n € Z, 


F(nAw) © VN [Fyfln 
where Aw = (NAx)7!. The error in this approximation is bounded by 


L?N 


In practice, this bound may significantly overestimate the actual error. 
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4. [BH95, pp. 181-188] If f is a continuous, piecewise smooth, periodic function with period p and 


Fourier series Douey Cpe n*, then 
1. 2s N N 
Cn © —=[Fyf]n RA eS 


JN 


where f is the Nth order sampling of f over [0, p] with spacing Ax = p/N. Moreover, if f is 
m-times differentiable on R and f‘”) is piecewise continuous and piecewise monotone for some 
m €N, then the error in this approximation is bounded by 


B 


Nm 


for some positive constant 6 independent of n and N. 


Examples: 
1. Let 


f= x fr 5k/3 


k=—00 


be the periodic array with index period N = 4 and coefficients 


fo=0, i=l, fpo=2, and fp=l. 


The Fourier transform of f is then the periodic array 


fore) 
F= S- Fidnaw 


n=—OO 


with index period 4, spacing Aw = 3/4, and coefficients given by 


Fo 1 1 J 4 0 6 

F a 3 }1 -1 1 12 1 1 | 3 
P = Awy/4 Fif = = tae 

PF 4/1] —-] 1 -l 2 210 

P; 1 a -l i 1 —3 


2. The 8th order sampling with spacing Ax = 1/6 of 


x forO<x <1/2 
f(x)=<1—-x forl/2<x<1 
0 otherwise 
isf=[fo, fi, fa. ---» f7]" with 
k/6 for k = 0, 1, 2, 3 
h=t(Z) = 1—k/6 fork =4, 5,6 
0 fork =7 


That is, 
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Multiplying this by the spacing Ax = 1/6 yields the scaled sampling 


T 
0, 0 


12 3 2 1 


f= 0, > > > > b 
36° 36° 36” 36” 36 


Letting F = F[ f], Aw = (NAx)"! = 3/4, and w = e?"/8, 


F(nAw) © V8 [Fafln 


1 
= a [Ow te lwit he 2w2n f 3yr" 4 2w'in 4 lwo 4 Ow" ae ow?" 


1 
=a [wo" + 2wo" +34 2w" + w7"| ; 


which simplifies to 
AD | 
F(nAqw) © V8 [Fel], = see [3 + 2 cos (=) + 4cos (=)| : 


The error bound on this approximation is 


j L?N 4+ 37|n| 
E,= 2 A = : 
(max | f’| + 27¢|n| ome t) Sap ie 
In fact, the Fourier transform of f is easily found to be 
sin(@/2) 2 
Flo) =e [SOL 
a0) 


So 


3n 


F(nAw) = F( 2%) = einen | 48inGnx/8) 2 
ne oe ; 


and the actual error in the approximation is 


En= |F(nde) mals [Fafln 
o4 [Seni 3 2cos (—*) soo (%)) 


For n = 0 through 4, the actual and approximate values of F (nAq) (to five decimal places) are 


1 
36 


F (0Aw) = 0.25000 V8 [Feflo = 0.25000 
F(1Aw) = — 0.108721. +i) V8 (F ef], = — 0.11448(1 + i) 
F (2Aq@) = 0.02252i /8 [Fefl2 = 0.02778: 

F (3A@) = 0.00207(1 — i) 8 [Fe]; = 0.00337(1 — i) 
F(4A@) = — 0.01126 /8 [Fafls = — 0.02778. 
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The corresponding error bounds and actual errors (to five decimal places) are 


Ey=0.11111 — ¢9 =0.00000 
E,; =0.37291 — €, = 0.00815 
E,=0.63471 — €) = 0.00526 
E;=0.89651 — ¢3 = 0.00183 
E,=1.15831 — ¢4 = 0.01652. 


Applications: 


1. Let f bea continuous, piecewise smooth function to be sampled that is nonzero only in the interval 
(0, 1) and which satisfies 


and |f'(x)| <1 VxeR. 


Ble 


|f(x)| < 


Consider the problem of approximating F(nAw) where F = F[f], Aw = 1/2, and n is any 
integer from 0 to 10. For each N €N, set 
1 2 2(N — 1) 


Ax = —— = — and L=(N-1)Ax= ; 
NAw N N 


and let f = [:fos fis a. Pere Rete be the corresponding scaled sampling of f, 


o~ 


Fr = f(kAx)Ax = f (=) = 


Then, for n = 0, 1, 2,..., 10, 


N-1 
2 ; 
F(nAw) © VN [Fryfln ae jf (= =) aoe 


with an error bound of 


L?N 4+10x 18 
= < : 
2(N — 1)2 2N N 


(max | f’| + 27 |n|Aq@ max | f|) 


2. [CGO00, p. 157] (Deconvolution) Assume g = f *h where g andh are known by Nth order samplings 
g and h, each with spacing Ax. The deconvolution formula for finding f, 


Fig] 


G 
ese = = 
f=F[V] where V = — = rh)’ 


is approximated by 


f(kdx) © VN [FV] 


where, letting Aw = (NAx)"1, V is the sequence given by 


Zs forn =0,1,2,..., N—-1. 
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This reduces to 


1 ve 
(kAx) © [Ary | 
f AxJ/N LN Jk 
where, letting G = ie ng and H = F, wh, 
Gr 
VY, =— forn=0,1,2,..., N-1. 
A, 


This requires, of course, that each H,, be nonzero. 


58.5 The Fast Fourier Transform 


A fast Fourier transform (FFT) is any of a number of algorithms for computing DFTs in which the number 
of arithmetic computations is greatly reduced through clever use of symmetries and cyclic structures in 
the DFT matrices. The FFT described here is the standard radix 2 FFT for computing the Nth order 
“alternative” discrete Fourier transform Dyv given by 


N-1 
[Duv]n = VNIF wv)n = So ve ON, 
k=0 


For additional details on implementing a radix 2 FFT and descriptions of other FFTs, see [BH95, 
chap. 10], [CG00], or [Wal96]. 


Algorithms: 
Two algorithms for computing the alternative DFT 
V= [Vo Vis Va... Vn-l? 
of an Nth order sequence 
v= [Vo V1, Vay «00> Yw—il)? 


are given. The first is the “first level” radix 2 FFT illustrating the basic concepts. The second is the more 
complete radix 2 FFT. 


1. (First level radix 2 FFT) This requires N = 2M for some M € N: 


* First, split vinto two Mth order sequences v° and v? composed, respectively, of the even-indexed 
and odd-indexed elements of v, 


E E E E E T T 
v S106 Ve RV ey Sess Vig_1! = [vo, V25 V4y e+e V2M-2] 
and 
O Oo O O Oo T T 
v SVE sev 5 Ves eke Vici = [v1, V3, V5y +005 V2om-11 


¢ Then compute the Mth order alternative DFTs 
[Mrs Vs ve ae Vircale = ve = Dyvt 
and 


[Vor, VP, Vee... Vid)? =V° = Dyv?. 
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2. 


Facts: 
All the 


Ts 


. For corresponding “fast” algorithms for computing the inverse DFT of v, replace w = e 
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* Then construct the Nth order sequence V = [V, Vi, Vo, ...» Vn—i]? from V® and V® via the 
butterfly relations 


VE yw “Vo. forn=M,M+1,M+4+2,...,N—-1 


7 VE +w"Vvo forn =0,1,2,...,M—1 


where w = e7?27/N, 
(Full radix 2 FFT) This requires N = 2? for some P € N. To simplify the description, let 
{0} ifk=0 
OLE ; ‘ 
{E,O} ifk=1,2,3,...,P 


* Recursively, split v into the sequences in the set 


{vm » K = 0,1,2,...,P} 


where v® = vand, letting © = o90102---ox and M = 2?-X-1, 
DE _ [, SE . SE SE Se] Pde oh x T 
VO ESV Wr st Vot wn hcey Mapeg| St |Vors Voc Vaca secy Vong 2p 
and 
20 — [20 yLO 1x0 vE0]" = [v2 zr yz yz ih 
Vo" = [Vo > Vp Va a ees Vg} [V9 V3 V5 9-229 Vom-i] - 


* For each of the 2? first order sequences in {v71"°? = [v™""9? ]} , set 


VP LYNG OP] fy MP2 OP] O12 OP 


¢ Set V = V™ where 
AVEIRO BIE OT Does PY 


is the set of sequences recursively constructed via the butterfly relations 


V= 


VE + wryXO forn=0,1,2,...,M—1 
VE — wt Myre forn=M,M+1,M+2,...,2M—1 


a sae -_ K 
where © = 090102-:-ox, M = 2?-*-!, and w = e(—?27/N)?" 


following facts except those with specific reference can be found in [BH95], [CG00], or [Wal96]. 


Let V be the sequence computed from an Nth order sequence v by either of the two algorithms 
above. Then V = Dyv. Multiplying V by N~'/? yields Fryv. 

—i2n/N and 
w= e(—i2n/N)2" | respectively, with w = e?2/N and w = e(2/N)2" in the above algorithms. 
[BH95, p. 393] Assume N = 2? »> 1 for some P € N. Using just the defining formulas, the 
computation of an Nth order DFT requires approximately 2N? arithmetic operations ((N — 1)? 
multiplications and N(N — 1) additions). Using the above described radix 2 FFT allows this same 
DFT to be computed with approximately (3.N/2) log, N arithmetic operations (N log, N additions 
and (N/2) log, N multiplications, plus another N multiplications by N~!”” ifit is desired to convert 
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the alternative DFT to the one previously described). Thus, the ratio of the number of arithmetic 
operations used to compute an Nth order DFT by the two methods is 


operations ifusingthe FFT — 3log, N 


operations if not using the FFT ~ AN 


4. Assume N = 2? > 1 for some P € N. Simply using the basic formulas, the computation of the 
discrete convolution of two sequences of order N, 


requires approximately 2. N? arithmetic operations ((N+1)N multiplies and (N — 1) N additions). 
On the other hand, this convolution can be computed using the convolution identity 


nN 


vew=F,' [VW] where V = Fyv and W = Fyw. 


Employing this identity along with the radix 2 FFT to compute the three DFTs involved requires 
approximately 


* 3 x (3N/2) log, N arithmetic operations for the DFTs, 


¢ Plus N multiplications for the product VW, 


for a total of approximately (9N/2) log, N arithmetic operations. The ratio of operations required 
to compute this convolution by the two methods is 


operations if using identities andthe FFT _ 9log, N 


operations if not using identities and the FFT ~ AN 


Examples: 
1. Consider finding the DFT of 


Y= [vo, V1, V2; v3)" = [1, 2, 3, 4]? 


using the full radix 2 FFT. The order here is N = 4 (so P = 2). Recursively splitting v into the even 
and odd index subsequences yields 


v' = [1,3]? and = v? = _[2, 4]7 
and then 
vee = [1], we? =[3], v0? =[2], and v?°? = [4]. 
(The oo = 0 superscript is being suppressed.) Setting V" = v'” yields 


vFF =[1], VF? =[3], VF =([2], and V°°? = [4]. 
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Applying the first round of butterfly relations (in which K = 1, M = 2?-k-l = 1landw = 


e(—i2m/4)2! =-1), 


Ve = Ve" —w'VyFo =l]-— (—1)°3 —-2 


Ve = ve + WV? =2+ (—1)°4= 6 


V0 = VOE — wi-1y0° = 2- (-1)%4=-2, 
yields the second order sequences 
vi =[4,-2]" and V°=[6,-2]". 


Applying the second (and last) round of butterfly relations (in which K = 0, M = 2?-K-1 = 2 


and w = e(-i27/4)2® _ _j), 


Vo= Vo twVi- = 44 (-i)°6 = 10 


Vi=Vi +w'V2 = -24 (-i)'(-2) = -2 + 2i 


WS Vw Se 6S 


V3 = Ve, —w? ?Vx2, = —2 - (-i)'(-2) = -2-2i, 
yields the alternative DFT 


V=DPv=[10, -24+2i, -2, -2-2i]’. 


2. Consider the computation of an Nth order DFT. When N = 1024 = 2!°, the ratio of operations 
required to compute this DFT with and without using the FFT is 


operations ifusingthe FFT | 3-10 


ae we ~7.3x 107°. 
operations if not using the FFT 4. 2!° 
When N = 11,048,576 = 2°, this ratio is 
operations if using the FFT 3-20 = 
~ ~14x 107. 


operations if not using the FFT ~ 4. 220 


3. Consider computing the convolution of two Nth order sequences. When N = 1024 = 21°, the ratio 
of operations required to compute this convolution with and without using the FFT and identities 
is 


operations if using identitiesandthe FFT — 9-10 


me 2.2 x 1077. 
operations if not using identities andthe FFT 4- 2!° 
When N = 11,048,576 = 279, this ratio is 
operations if using identities and the FFT 9-20 se 
~ 43x 10°. 


operations if not using identities and the FFT 4. 270 
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59.1 Introduction 


Linear algebra appears throughout physics. Linear differential equations, both ordinary and partial, appear 
through classical and quantum physics. Even where the equations are nonlinear, linear approximations 
are extremely powerful. 

Two big ideas underpin linear analysis in physics. The first is the Superposition Principle. Suppose we 
have a linear problem where we need to compute the output for an arbitrary input. If there is a solution to 
the problem with input J, and output Oj, a solution with input J, and output Oj, etc., then the response 
to the input c) Ij +--+ + cyl is cyO, +--+ cx Ox. It is, therefore, enough to solve our problem for a 
limited set of inputs J;,, as long as an arbitrary input can be written as a linear combination of these special 
cases. 

The second big idea is the Decoupling Principle. Ifa system of coupled differential equations (or difference 
equations) involves a diagonalizable square matrix A, then it is helpful to pick new coordinates y = [x], 
where B is a basis of eigenvectors of A. Rewriting our equations in terms of the y variables, we discover 
that the evolution of each variable y, depends only on yz, and not on the other variables, and that the 
form of the equation depends only on the kth eigenvalue of A. We can then solve our equations, one 
variable at a time, to get y as a function of time and, hence, get x as a function of time. (When A is not 
diagonalizable, one uses a basis for which [A]g is in Jordan canonical form. The resulting equations for y 
are not completely decoupled, but are still relatively easy.) 

Thanks to Newton’s Law, F = ma, much of classical physics is expressed in terms of systems of second 
order ordinary differential equations. If the force is a linear function of position, the resulting equations are 
linear, and the special solutions that come from eigenvectors of the force matrix are called normal modes 
of oscillation. For nonlinear problems near equilibrium, the force can always be expanded in a Taylor 
series, and for small oscillations the leading (linear) term is dominant. Solutions to realistic nonlinear 
problems, such as small oscillations of a pendulum, are then closely approximated by solutions to linear 
problems. 
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Linear field equations also permeate classical physics. Maxwell’s equations, which govern electromag- 
netism, are linear. There are an infinite number of degrees of freedom, namely the value of the field 
at each point, but the Superposition Principle and the Decoupling Principle still apply. We use a con- 
tinuous basis of possible inputs, namely Dirac 6 functions, and the resulting outputs are called Green’s 
functions. The response to an arbitrary input is then the convolution of the input and the relevant Green’s 
function. 

Nonrelativistic quantum mechanics is governed by Schrédinger’s equation, which is also linear. Much 
of quantum mechanics reduces to diagonalizing the Hamiltonian operator and applying the Decoupling 
Principle. 

Symmetry plays a big role in quantum mechanics. Both vectors and operators decompose into represen- 
tations of the rotation groups SO(3) and SU(2). The irreducible representations are finite-dimensional, 
so the study of rotations (and angular momentum) often reduces to a study of finite matrices. 


59.2. Normal Modes of Oscillation 


Suppose we have two blocks, each with mass m, attached to three springs, as in Figure 59.1, with the spring 
constants as shown, and let x;(t) be the displacement of the 7th block from equilibrium at time f. It is easy to 
see that if x,(0) = x2(0), and if x, (0) = x2(0), then x; (t) = x(t) for all time. The middle spring never gets 
stretched, and the two blocks oscillate, in phase, with angular frequency w; = ./k,/m. If x)(0) = —x,(0) 
and x2(0) = —x,(0), then by symmetry x2(t) = —x,(t) for all time, and each block oscillates with angular 
frequency @2 = \/ (ki + 2k2)/m. (This example is worked out in detail below.) These two solutions, with 
x(t) = +x2(t), are called normal modes of oscillation. Remarkably, every solution to the equations of 
motion is a linear combination of these two normal modes. 


Definitions: 

Suppose we have an arrangement of blocks, all of the same mass m, and springs with varying spring con- 
stants. Let x) (tf), ...,Xn(t) denote the locations of the blocks, relative to equilibrium, andx = [x1,..., Xn] 2. 
For any function f(t), let f(t) = df/dt. The kinetic energy is T = m 5~, x{/2. The potential energy is 


MMA 


SSMS 


FIGURE 59.1 Coupled oscillators. 
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V@™)=y; j Vij XiXj /2, where A = (aj;) is a symmetric matrix. The equations of motion are 


d’x 
Let B = {z),...,Zn} be a basis of eigenvectors of A, and let y(t) = [x(t)]g be the coordinates of x(t) in 
this basis. 
Facts: 


(See Chapter 13 of [Mar70], Chapter 6 of [Gol80], and Chapter 5 of [Sad01].) 


1. A is diagonalizable, and the eigenvalues of A are all real. 
2. The eigenvectors can be chosen orthonormal with respect to the standard inner product: (z;,z;) = 


Oj. 
ij 
3. The initial conditions for y(t) can be computed using the inner product: yx(0) = (zz, x(0)), 7.(0) = 
(Zk; x(0)) . 
4. In terms of the y variables, the equations of motion reduce to md’ y,/dt? = —Axyg, where Ax is 


the eigenvalue corresponding to the eigenvector zx. 
5. The solution to this equation depends on the sign of Ay. IfA, > 0, set wy, = \/Ax/m. We then have 


ye(0) 
x(t) = yx(0) cos(eoxt) + 7 — sin(wit) 
If Ax < 0, set Ky = /—A,/m and we have 
y.(0 
ye(t) = ye(0) cosh(xyt) + ~ M ) ing): 
k 


Finally, if A, = 0, then 
ye(t) = yx(O) + yx (ODE. 


6. If the system has translational symmetry, then there is a A = 0 mode describing uniform motion 
of the system. 

7. Ifthe system has rotational symmetry, then there is a 1 = 0 mode describing uniform rotation. 

8. All solutions of the equations of motion are of the form x(t) = S~ y;,(t)z,, where for each nonzero 
As Ve(t) is of the form given in Fact 5. 


Examples: 


1. In the block-and-spring example above, the kinetic energy is m(x? + x3)/2, while the potential 
kth —k . 
ae es ) . The eigen- 
values of A are k, and k, +2k), with normalized eigenvectors (/2/2, 2/2) T and (/2/2, —/2/2) i 
Both eigenvalues are positive, so we have oscillations with angular frequencies @; = /k,/m and 
@) = \/(k, + 2k)/m. Suppose we start by pushing the first block to the right and letting go. That 
is, suppose x(0) = (1,0)? and x(0) = (0,0). From the initial data we compute 


energy is (ky.x? + kp (x1 — x2)? +k, x3) /2 = (x, Ax) /2, where A = ( 
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yi (0) = ? (ay (0) + 90 (0)) = ¥2/2 
y2(0) = 2 (ay (0) — a x2(0)) = /2/2 
yi (0) = Y? (a(0) + X2(0)) = 
yi (0) = 2? (a(0) — X2(0)) = 


yi(0) 


Q) 


yi(t) = yi (0) cos(@ ft) + sin(@,t) = V2. cos(a)t)/2 


0) sin(@2t) = V2 cos(wot) /2 
@2 


1 Gales + cos(wt) ) 


2 \ cos(@,t) — cos(@t) 


y2(t) = y2(0) cos(w@2t) + 


> 


x(t) = yi (t)z) + y2(t)z = 


2. LC circuits obey the same equations as blocks and springs, with the capacitances playing the role 


of spring constants and the inductances playing the role of mass, and with the current around each 
loop playing the role of x;. 

Small oscillations: A particle in an arbitrary potential V(x), or a system of identical-mass particles 
in an arbitrary n-body potential, follows the equation md*x/dt? = —V V(x). Ifx = Xp isa critical 
point of the potential, so V V(xo) = 0, then we expand V(x) around x = xo ina Taylor series: 


V(x) = V0) + 5 So a(x = my) i= 29); + Ola 201°) 


where ajj = aa ,so VV(x) = A(x— xy) + O(|x —xo|”), and our displacement x — xy from 
i°*7 |x=xo 
equilibrium is governed by the approximate equation md?(x — xq)/dt? = — A(x — x). 


For example, a pendulum of mass m and length ¢ has quadratic kinetic energy m€?6?/2 and 
nonlinear potential energy mg €(1 — cos(@)). For 6 small, this potential energy is approximated by 
mg 67/2, and the equations of motion are approximated by €d?6/dt? = —g@, and yields oscilla- 
tions of angular frequency ./g/€. The same ideas apply to motion ofa pendulum near the top of the 
circle: 9 = 2. Then V(@) © mg£(2— (6 — 1)” /2), and our equations of motion are approximately 
ed?(@—x)/dt? =+ (0 —s). The deviation of 6 from the unstable equilibrium grows as e*‘, with 
Kk = /g/€, until 6 —z is large enough that our quadratic approximation for V(@) is no longer valid. 

Finally, one can consider two pendula, near their stable equilibria, attached by a weak spring. 
The resulting equations are almost identical to those of the coupled springs of Figure 59.1. 
Central force motion (see Chapter 3 of [Gol80] or Chapter 8 of [Mar70]): In systems with symmetry, 
it is often possible to use conserved quantities to integrate out some of the variables, obtaining 
reduced equations for the remaining variables. For instance, if an object is moving in a central force 
(e.g., a planet around a star or a classical electron around the nucleus), conservation of angular 
momentum allows us to integrate out the angular variables and get an equation for the distance r. 
The radius then oscillates in a pseudopotential V(r), obtained by adding a 1/r? centrifugal term 
to the true potential Vo(r). Orbits that are almost circular are described by small oscillations of the 
variable r around the minimum of the pseudopotential V(r), and the frequency of oscillation is 
\/ V" (ro)/m, where the pseudopotential has a minimum at r = ro. When the true potential is a 
1/r attraction (as with gravitation and electromagnetism), these oscillations have the same period 
as the orbital motion itself. Planets traverse elliptical orbits, with the sun at a focus, and the nearest 
approach to the sun (the perihelion) occurs at the same point each year. When the true potential is 
an r? attraction (simple harmonic motion), the radial oscillations occur with frequency twice that 
of the orbit. The motion is elliptical with the center of force at the center of the orbit, and there are 
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two perihelia per cycle. For almost any other kind of force, the radial oscillations and the rotation 
are incommensurate, the orbit is not a closed curve, and the perihelion precesses. 


59.3 Lagrangian Mechanics 


In the previous section, we assumed that all the particles had the same mass or, equivalently, that the 
kinetic energy was proportional to the squared norm of the velocity. Here we relax this assumption and 
we also allow generalized coordinates. 


Definitions: 


The Lagrangian function is L(q,q) = T — V, where T is the kinetic energy and V is the potential energy. 
One can express the Lagrangian in terms of arbitrary generalized coordinates q and their derivatives q. 
The kinetic energy is typically quadratic in the velocity: T = (q, B(q)q)/2, where the symmetric “mass 
matrix” B may depend on the coordinates q, but not on the velocities q. The potential energy V depends 
only on the coordinates q (and not on q), but may be nonlinear. 

If qo is a critical point of V, we consider motion with q close to qo and q small. 


Facts: 
(See Chapter 7 of [Mar70] or Chapters 2 and 6 of [Gol80].) 


d ()== 
dt \ dq.) 9qx 


reduce to the approximate equations of motion 


1. The Euler-Lagrange equations 


d’(q — q) 
B =-A ; 
qe (q — q) 
where aj; = oe , essentially as before, and the mass matrix B is evaluated at q = qo. Instead 
oF \q=qo 


of looking for eigenvalues and eigenvectors of A, we look for numbers A; and vectors z, such that 
AZy = Ax Bzy. (See Chapter 43.) We then let y = [q — qol]z. 

2. The matrices A and B are symmetric, and the eigenvalues of B are all positive. 

3. The numbers A, are the roots of the polynomial det(x B — A). When B is the identity matrix, these 
reduce to the eigenvalues of A. 

4, One can find a basis of solutions z; to Azy = Azz, with A; real. The numbers A, are the eigenvalues 
of B~'A, or equivalently of the symmetric matrix B~'/?AB~'/?, which explains why the A;’s 
are real. 

5. The eigenvectors can be chosen orthonormal with respect to an inner product involving B. (See 
Chapter 5.) That is, if (u, v) 3 = u’ By, then (z;,z;) 3 = 5jj. 

6. The initial conditions for y(t) can be computed using the modified inner product of the previous 
fact: ye(0) = (2k, q(0) — qo(0)) 5 je(0) = (ze, 4(0)) n- 

7. Interms of the y variables, the approximate equations of motion reduce to the decoupled equations 
a? y,/dt? = —Agyp. 

8. The solution to these equations depends on the sign of Ax. If Ax > 0, set m, = VSAxs if Ax < 0, 
set Ky = ./—A,. With values of w; or Kz, these the solutions take the same form as in Fact 5 of 
section 59.2. 

9. If the system is symmetric under the action of a continuous group, then there is a A = 0 mode for 
each generator of this group. 
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FIGURE 59.2 A double pendulum. 


10. All solutions of the approximate equations of motion are of the form q(t) = qo +). yg(t)z%, where 
for each nonzero Ax, yx(t) is of the form given by Fact 8 above (and Fact 5 of Section 59.2). 


Examples: 


1. Consider the double pendulum of Figure 59.2, where each ball has mass m and each rod has 
length €. For large motions, this system is famously chaotic, but for small oscillations it is sim- 
ple. The two coordinates are the angles 6, and @,, and the potential energy of the system is 


mgl(3 — 2cos(O,) — cos(@2)) © mgl(O? + 63/2), s0 A = mg (; 2 The kinetic energy 
is _ (6? + (sin(6;)6, + sin(62)@)? + (cos(@)6, + cos(@)6>)*). For small values of 6; and 6, 

1): det(xB — A) = m’€4*(x? — 
4(g¢/€)x + 2g7/€7), with roots Ay = (g¢/£)(2 + /2) and A, = (g/€)(2 — /2), and with z; = 
ci(1, +/2)', i = 1,2, where c; are normalization constants. The two normal modes are as follows: 


There is a fast mode, with w, = 1/ (g/€)(2 + J/2) & 1.8478,/¢/€, with the two pendula swinging 


in opposite directions, and with the bottom pendulum swinging V2 more than the top; there is a 
slow mode, with w1/ (g/£)(2 — /2) & 0.76544/ g/€, with the two pendula swinging in the same 
direction, and with the bottom pendulum swinging ./2 more than the top. 


this is approximately me" (26? + 63 + 26,4), s0 B = me? ( 


59.4 Schrédinger’s Equation 


In quantum mechanics, the evolution of a particle of mass m, moving in a time-dependent potential 
V(x, t), is described by Schrédinger’s equation, 


aw(x, t) 
ot 


ih 


i 2 
2m 


where fi is Planck’s constant divided by 277, and the square of the complex wavefunction (x, t) describes 
the probability of finding a particle at position x at time f. Space and time are not treated on equal footing. 
We consider the wavefunction w to be a square-integrable function of x that evolves in time. 
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Definitions: 


Let H = L?(R") be the Hilbert space of square-integrable functions on R” with “inner product” 


hh) = [ Fava"s, 


Note that this inner product is linear in the second factor and conjugate-linear in the first factor. Although 
mathematicians usually choose their complex inner products (u, v) to be linear in u and conjugate-linear 
in v, among physicists the convention, and notation, is invariably that of the above equation. The bracket 
of g and y can be viewed as a pairing of two pieces, the “bra” (@| and the “ket” |y). The ket |y) is a vector 
in H, while (| is a map from H to the complex numbers, namely, “take the inner product of @ with an 
input vector.” 

The Hermitian adjoint of an operator A, denoted A*, is the unique operator such that (A*¢|w) = 
(p| Ay), for all vectors , y. An operator A is called Hermitian, or self-adjoint, if A* = A, and unitary 
if A* = AT}, 

The commutator of two operators A and B, denoted [A, B], is the difference AB — BA. Aand B are 
said to commute if AB = BA. 

The expectation value ofan operator A in the state | y) is the statistical average of repeated measurements 
of A in the state |y), and is denoted (A), with the dependence on |) implicit. The uncertainty in A, 
denoted A A, is the root mean squared variation in measurements of A. 

The generalized eigenvalues of an operator A are points in the spectrum of A, and the generalized 
eigenvectors are formal solutions to Aly) = A|y). These may not be true eigenvalues and eigenvectors 
if w is not square integrable. (See Facts 11 to 15 below.) This use of the term “generalized eigenvector,” 
which is standard in physics, has nothing to do with the same term in matrix theory (where it signifies 
vectors v for which (A — AI)*v = 0 for some positive integer k). 


Facts: 
(See Chapter 6 of [Sch68] or Chapters 5 and 7 of [Mes00].) 


1. The Schrédinger equation can be recast as an ordinary differential equation with values in 1: 


lw) 

h— = H(t : 

ins (Iv) 
where H = — mye + V is the Hamiltonian operator. 


2. Physically measurable quantities, also called observables, are represented by Hermitian operators. It 
is easy to see that the position operator (Xy)(x) = x(x), the momentum operator (Py)(x) = 
—ihV w(x), and the Hamiltonian H = P?/2m + V are all self-adjoint. 

3. If an observable a is represented by the operator A, then the possible values of a measurement of 
a are the (generalized) eigenvalues of A. 

4. Two Hermitian operators A, B can be simultaneously diagonalized if and only if they commute. 

5. Suppose the state of the system is described by the vector |) = >, Cnlén), where > [cn |? = 1 
and each |@,,) is a normalized eigenvector of A with eigenvalue A,,. Then the probability of a 
measurement of a yielding the value A, is |cn|’. 

6. If |y) is as in the previous Fact, then the expectation value of A is (A) = >, Anleul?. 

7. The uncertainty of A satisfies (A.A)* = (A?) — (A)?. 

8. If A, B, and C are Hermitian operators with [A, B] = iC, then AAAB > |(C)|/2. 

9. In particular, XP — PX = ih,so AXAP > h/2. This is Heisenberg’s uncertainty principle. 

10. If the Hamiltonian operator does not depend on time, then energy is conserved. In fact, if 
\v(0)) = Xenlbn), where Hipn) = Enlon), then |W(t)) = Yo cne?="/"|g,,). (Eigenvalues of 
the Hamiltonian are usually denoted E,,, for energy.) Solving the Schrodinger equation is tanta- 
mount to diagonalizing the Hamiltonian and using a basis of eigenvectors. 
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Operators may have continuous spectrum, in which case the generalized eigenvectors are not 
square-integrable. In particular, e’** is a generalized eigenvector for P = —ihd/dx with general- 
ized eigenvalue hk, and the Dirac delta function 6(x — a) is a generalized eigenvector for X with 
eigenvalue a. 

Let |A,a) be a generalized eigenvector of the operator A with generalized eigenvalue a. If A has 
continuous spectrum, then the decomposition of a state |v) involves integrating over eigenvalues 
instead of summing: |) = [ f(a)|A,a)da. The generalized eigenstates are usually normalized 
so that (wily) = f | f (a) |?do. Equivalently, (A,a@|A, 6) = (a — B). 

For continuous spectra, | f ()|? is not the probability of a measurement of A yielding the value 
a. Rather, | f(@)|* is a probability density, and the probability of a measurement yielding a value 
between a and bis [’ | f(a)|2da. 

The two most common expansions in terms of generalized eigenvalues are for the position operator 
and the momentum operator: |W) = f w(x)|X,x)dx = [ wh (k)|P, hk)dk. The coefficients (k) 
of |yv) in the momentum basis are the Fourier transform of the coefficients w(x) in the position 
basis. From this perspective, the Fourier transform is just a change-of-basis. 

An operator may have both discrete and continuous spectrum, in which case eigenfunction ex- 
pansions involve summing over discrete eigenvalues and integrating over continuous eigenvalues. 
For example, for the Hamiltonian of a hydrogen atom, there are discrete negative eigenvalues that 
describe bound states, and a continuum of positive eigenvalues that describe ionized hydrogen, 
with the electron having broken free of the nucleus. 


Examples: 


1. 


The one dimensional harmonic oscillator. We have seen that a classical harmonic oscillator with 
potential energy kx?/2 has frequency w = ./k/m, so we write the Hamiltonian of a quantum 
mechanical harmonic oscillator as 


P2 Z kX? PP? + mw? X? 
2m 2 2m : 


We will compute the eigenvalues and eigenvectors of this Hamiltonian. 
We define a lowering operator 


P —imoxX 
J2mho ~ 


The Hermitian conjugate of a is the raising operator 


a= 


= P+imwX 
7 /2mhw : 


Note that a and a* do not commute. Rather, [a,a*] = 1. In terms of a and a*, the Hamiltonian 
takes the form 


a* 


h h 
H = ha(a*a + aa*)/2= 5 (2ata F1j= 5 (2aa" 1). 


Note that a*a is positive-definite, since (|a*aw) = (ay|aw) > 0, so the eigenvalues of energy 
must all be at least hw/2. 

Since Ha = a(H — hw), the operator a serves to lower the energy of a state by hw. If |¢) 
is an eigenvector of H with eigenvalue E, then Ha|@) = a(H — hw)|¢) = a(E — hw)|¢) = 
(E — hw)a|@), so a|@) is either the zero vector or a state with energy E — hw. Since we cannot 
reduce the energy below hiw/2, by applying a repeatedly (say, n > 0 times), we must eventually get 
a vector |¢o) for which al¢y) = 0. But then H|¢o) = fe (2a*a + 1)|¢o) = Pe | bo). Since it took n 
lowerings to get the energy to hw/2, our original state |p) must have had energy (n + })ho. 
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The notation |7) is often used for this nth excited state, so a|n) isa constant times |n—1). It remains 
to compute that constant. Normalizing (n|n) = 1 and picking the phases such that (n — 1|an) > 0, 
we compute (an|an) = (n\a*an) = (nl Hin) 5 = n(n|n) = n, so a|n) = ./n|n — 1). A similar 
calculation yields a*|n) = ./n + I|n + 1) and, hence, |n) = (a*)"|0)//n!. 


The state |0) is in the kernel of a. In a coordinate basis, where X is multiplication by x and 


P = —ihd/dx, the equation a|0) = 0 becomes a first-order differential equation 
d 
w(x) i Ucar ~0, 
dx h 


whose solution is the Gaussian w(x) = exp(—mwx?/2h) times a normalization constant. The nth 
MX 


state is obtained by applying the differential operater 4. — “=~ to the Gaussian n times. The result 
is an nth order polynomial in x times the Gaussian. 


59.5 Angular Momentum and Representations 
of the Rotation Group 


The same techniques that solved the harmonic oscillator also work to diagonalize the angular momentum 
operator. 


Definitions: 

Angular momentum is a vector: L=Xx P, or in coordinates, L; = X7P3— X3P), L, = X3P, — X,P3, 
Ls = X,P) — XP). Each L; is a self-adjoint observable. We define L? = L? + L} + L3, and define a 
raising operator L, = L, + iL, anda lowering operator L_ = L, — iL». 


Facts: 
(See Chapter 7 of [Sch68] or Chapter 13 of [Mes00].) 


1. The three components of angular momentum do not commute. Rather, 
[L,,L2] = ihLs, [L2,L3] =ihlLy, [L3,L)] =ihLp. 


By the uncertainty principle, this means that only one component of the angular momentum can 
be known at a time. 

2. L? is Hermitian, and each [L;,L?] = 0. It is possible to know both L? and L3, and we consider 
simultaneous eigenstates |¢,m) of L? and L3, where & labels the eigenvalue of L? and hm is the 
eigenvalue of L3. 

3, [L?, La] = Oand [L3, Li] = +hL 4. This means that L , does not change the eigenvalue of L?, 
but increases the eigenvalue of L3 by h. Likewise, L_ decreases the eigenvalue of L3 by h. 

4. Since L? — L} = L? + L4 > 0, there is a limit to how big m (or —m) can get. For each £, there is 
a state |, max) for which L+|£, max) = 0, and a state |, mmin) for which L_|£, Minin) = 0. We 
set the label £ to be equal to Mmnax. 

5. L-Ly=L?4+1$—hAL3andL,L_ = 1} +1} +AL;, so we can write L? in terms of Ls and L;: 
P27 =L_Ly + le+hl3 =LyL_+15—-AhL3. 

6. The minimum value of m is —€. Since 20 = mmax — Mmin is an integer, € must be half of a 
nonnegative integer. 

7. The states |€,m), with m ranging from —£ to @, form a (2@ + 1) dimensional irreducible repre- 
sentation of the Lie algebra of SO(3). We denote this representation V;, and call it the “spin-£” 
representation. 

8. In Ve, we have L?7|/€,m) = hre(€ + 1)|€,m), L3|€,m) = mhlé,m), and Lalé,m) = 
h/l(E + 1) — mm £ 1)|l,m + 1). 
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9. If u is a unit vector, then a rotation by the angle @ about the u axis is implemented by the unitary 
operator exp(—i0L - u/h). 

10. Since rotation by 27 equals the identity, representations of the Lie group SO(3) satisfy the additional 
condition exp(27iL3) = 1, which forces m (and, therefore, £) to be an integer. 

11. If one particle has angular momentum ¢, and another has angular momentum ¢), then the com- 
bined angular momentum can be any integer between |¢; — £2| and £; + £2. In terms of represen- 
tations, Vy, ® Ve, = Or _4 Ve- 

12. The Lie group SU(2) is the double cover of SO(3), and has the same Lie algebra. The generators 
are usually denoted J rather than L, and the maximum value of m is denoted j rather than £, but 
otherwise the computations are the same. J describes the total angular momentum of a particle, 
including spin, and j can be either an integer or a half-integer. 

13. Particles with j integral are called bosons, while particles with j half-integral are called fermions. 

14. If the Hamiltonian is rotationally symmetric, then angular momentum is conserved, and our 
energy eigenstates can be chosen to be eigenstates of J? and J3. 


59.6 Green’s Functions 


Expansions in a continuous basis of eigenfunctions are not limited to quantum mechanics. The Dirac 6 is 
an eigenfunction of position, and any function can be written trivially as an integral over 5 functions: 


fx) =f fnv~- dy = f FONIXyay. 


It, therefore, suffices to solve linear input-output problems in the case where the input is a 6-function 
located at an arbitrary point y. The resulting solution G(y, x) is called a Green’s function (or in some 
texts, Green function) for the problem, and the solution for an arbitrary input f(x) is the convolution 


J FO)GU,x)dy. 


Facts: 


(See [Jac75], especially Chapters 1 to 3, for many applications to electrostatics, and see Chapter 11 of 
[Sad01] for a general introduction to Green’s functions.) 


1. Green’s functions are sometimes called integral kernels, especially in the mathematics literature, or 
propagators in quantum field theory. The term propagator is also sometimes used for the Fourier 
transform of a Green’s function. 

2. Linear partial differential equations appear throughout physics. Examples include Maxwell’s equa- 
tions, Laplace’s equation, Schrédinger’s equation, the heat equation, the wave equation, and the 
Dirac equation. Each equation generates its own Green’s function. 

3. Some boundary value problems involve Neumann boundary conditions, in which the normal 
derivative of a function (as opposed to the value of a function) is specified on S, and some prob- 
lems involve mixed Neumann and Dirichlet conditions. The formalism for these cases is a simple 
modification of the Dirichlet formalism. 

4. Two common techniques for computing Green’s functions are Fourier transforms and the method 
of images. 

5. Fourier transforms apply when the problem has translational symmetry, as in the heat equation 
example, above. We decompose a 4 function as a linear combination of exponentials e‘** 
the response for each exponential, and re-sum. 

6. The method of images is illustrated in Example 2, where the actual response G(y, x) is a sum of 


, compute 


two terms. The first is the response Go(y, x) to the actual charge at y, computed without boundary, 
and the second is the response to a mirror charge, located at a point outside D, and chosen so that 
the sum of the two terms is zero on S. 
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Examples: 


1. Electrostatics without boundaries. The electrostatic potential p(x) is governed by Poisson’s equation: 
Vp = — Anp(x), 


where p(x) is the charge density. Here, p is the input and ¢ is the output. Since the solution to 
V’G(y,x) = —4767(x — y) is G(y,x) = |x — y|71, the potential due to a charge distribution 
p(x) iso(x) = f d* yo(y)/|x — y|. (Note that, when we write V?G(y, x), we are taking the second 
derivative of G(y, x) with respect to x. The variable y is just a parameter.) 

2. Electrostatics with boundary conditions. Poisson’s equation on a domain D with boundary S is 
more subtle, as we need to apply boundary conditions on S. Suppose that D is the exterior of a 
ball of radius R, and that we apply the homogeneous Dirichlet boundary condition @ = 0 on S. 
(This corresponds to S being a grounded conducting sphere.) The function Go(y, x) = 1/|x — y| 
satisfies V?Go(y, x) = —476?(x — y), but does not satisfy the boundary condition. The function 
G(y,x) = Goly,x) — pi Go(R7y/Iyl?s x) satisfies V?G(y, x) = —4767(x — y) and G(y,x) = 0 
forx € S. 

Nonzero boundary values can be considered part of the input. If we want to solve the equation 
V’¢ = —4zp on D with boundary values f(x) on S, then we have two different Green’s functions to 
compute. For each y € D, we compute G;(y, x), the solution to V?G,(y, x) = —4763(x — y) with 
boundary value zero on S. For each z € S, we compute G2(z, x), the solution to V’?G2(z,x) =0 
on D with boundary value 6?(x — z) on S. Our solution to the entire problem is then ¢(x) = 
Ip PyGily,x)o(y) + J, d?zf (2) Galz, x). 

3. The heat kernel. In R2, with variables x and t, let D be the region t > 0, so S is the x-axis. We look 
for solutions to the heat equation 


with boundary value f(x,0) = fo(x). Since G(y, x,t) = exp(—(x — y)*/4t)/./4zt is a solution 
to (3) and approaches 6(x — y) as t — 0, the solution to our problem is 


fED= / 6G xO,G)dS = i} exp(—(x — y)?/4t) foly)dy. 
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60.1 Introduction 


Biomolecular modeling is an active research area in computational biology. It studies the structures and 
functions of biomolecules by using computer modeling and simulation [Sch03]. Proteins are an important 
class of biomolecules. They are encoded in genes and produced in cells through genetic translation. Proteins 
are life supporting (or sometimes, destructing) ingredients (Figure 60.1) and are indispensable for almost 
all biological processes [Boy99]. In order to understand the diverse biological functions of proteins, the 
knowledge of the three-dimensional structures of proteins is essential. Several structure determination 
techniques have been used, including x-ray crystallography, nuclear magnetic resonance spectroscopy 
(NMR), and homology modeling. They all require intensive mathematical computing, ranging from data 
analysis to model building [Cre93]. 

As in all other types of scientific computing, linear algebra is one of the most powerful mathematical 
tools for biological computing. Here we review several subjects in biomolecular modeling, where linear 
algebra has played a major role, including mapping from distances to coordinates in NMR structure 
determination (Section 60.2), solving the Procrustes problem for structural comparison (Section 60.3), 
exploiting the structure of the Karle-Hauptman matrix in protein x-ray crystallography (Section 60.4), 
computing the fast and slow modes of protein motions (Section 60.5), and solving the flux balancing 
equations in metabolic network simulation (Section 60.6). The last subject actually involves the modeling 
of a large biological system, something beyond conventional biomolecular modeling, yet of increased 
research interests in computational systems biology [Kit99]. 
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FIGURE 60.1 Example proteins: Humans have hundreds of thousands of different proteins (e.g., hemoglobin protein, 
1BUW, in blood in 1a) and would not be able to maintain normal life even if short of a single type of protein. On 
the other hand, with the help of some proteins (e.g., protein, 2PLV, supporting poliovirus in 1b), viruses are able to 
grow, translate, integrate, and replicate, causing diseases. Some proteins themselves are toxic and even infectious such 
as the proteins in poisonous plants and in beef causing the Mad Cow Disease (e.g., prion protein, 114M-D, in human 
in Ic). 


60.2 Mapping from Distances to Coordinates: NMR Protein 
Structure Determination 


A fundamental problem in protein modeling is to find the three-dimensional structure of a protein and 
its relationship with the protein’s biological function. One of the experimental techniques for structure 
determination is to use the nuclear magnetic resonance (NMR) to obtain some information on the distances 
for certain pairs of atoms in the protein and then find the coordinates of the atoms based on the obtained 
distance information. Mathematically, the second part of the work requires the solution of a so-called 
distance geometry problem, i.e., determine the coordinates for a set of points in a given topological space, 
given the distances for a subset of all pairs of points. We consider such a problem with the distances for all 
pairs of points assumed to be given. 


Definitions: 


The coordinate vector for atom i is a vector xj = (Xj,1, Xi2, xi3)', where Xj,1, Xj,2, and x;,3 are the first, 
second, and third coordinates of atom i, respectively. 


The distance between atoms i and j is defined as d;,; = ||x; — x;||, where x; and x; are coordinate 
vectors of atoms 7 and j, and || - || is the Euclidean norm. 
The coordinate matrix for a protein is a matrix of coordinates denoted by X = {xj,; :i = 1,...,n, 


j = 1,2,3}, where nis the total number of atoms in the protein, and row i of X is the coordinate vector 
of atom i. 

The distance matrix for a protein is a matrix of distances denoted by D = {dj,j:i,j =1,..., n}, where 
d;,; is the distance between atoms i and j. 

The problem of computing the coordinates of atoms (X) given a set of distances between pairs of atoms 
(D) is known as the molecular distance geometry problem. 


Facts: 


1. [Sax79] If the protein structure and, hence, X are known, D can immediately be computed from 
xX. Conversely, if D is known or even partially known, X can also be obtained from D, but the 
computation is not as straightforward. The latter is proved to be NP-complete for arbitrary sparse 
distance matrices. 
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2. [Blu53] Choose a reference system so that the origin is located at the last atom, or in other words, 
Xn = (0, 0,0)7. Let X be a submatrix of X, X° = {x,j: i=1,...,.0—-1, j =1, 2, 3}, and D 
be a matrix derived from D, D° = {(d?,, — d?. + d;,,)/2 : i,j =1,...,n—1}. Then, matrix D’ 
is maximum rank 3 and X X7=D.. 

3. [CH88] Let D’ = UXUT be the singular-value decomposition of D’, where U is an orthogonal 
matrix and © a diagonal matrix with the singular values of D’ along the diagonal. If D’ is a matrix 
of rank less than or equal to 3, the decomposition can be obtained with U being (nm — 1) x 3 and 
x being 3 x 3, and X =U" solves the equation XxXTH=D. 


Algorithm 2: Computing Coordinates from Distances 

Given an n x n distance matrix D, 
1. Compute D° = {(d?,, — d?. + d;,,)/2 :i,j =1,...,n—- 1}. 
2. Decompose D° = UXU™ to obtain X°U[1: n—1,1:3]D/?[1:3,1:3]. 
3. X[1:n—1,1:3] = X°[1:n—1,1:3], X[n1:3] = [0,0,0]. 


Examples: 


1. Given the distances among four atoms, D, determine the coordinates of the atoms, X, where 


Following Algorithm 1, 


1 0 0 
D°=|0 1 0 
0 0 1 


Compute the singular value decomposition of D’. Obviously, D) = UDU™, with 


100 100 
U=]|0 1 0!\|, O=|01 0 
001 001 
Then, 
1 0 0 
x°=|0 1 0 
0 0 0 
and 
100 
010 
X= 
001 
000 
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FIGURE 60.2 3D structures of protein 1HMV p66 subunit: The structure on the left was determined by x-ray 
crystallography, while on the right by solving a distance geometry problem given the distances for all the pairs of atoms. 
The RMSD for the two structures when compared on all the atoms is around 1.0e-04 A. (Photo courtesy of Qunfeng 
Dong.) 


2. Figure 60.2 shows two 3D structures of the p66 subunit of the HIV-1 retrotranscriptase (LHMV), 
one determined experimentally by x-ray crystallography [RGH95] and another computationally 
by solving a molecular distance geometry problem using the SVD method with the distance data 
generated from the known crystal structure. The RMSD (see description in section 60.3) for the two 
structures when compared on all the atoms is around 1.0e—04 A, showing that the two structures 
are almost identical. 


60.3 The Procrustes Problem for Protein Structure Comparison 


The structural differences between two proteins can be measured by the differences in the coordinates 
of the atoms for all corresponding atom pairs. The comparison is often required for either structural 
validation or functional analysis. The calculation can be done by solving a special linear algebra problem 
called the Procrustes problem [GL89]. 


Definitions: 


Let X and Y be two n x3 coordinate matrices for two lists of atoms in proteins A and B, respectively, where 
X; = (Xj,1, X12, Xj,3) is the coordinate vector of the ith atom selected from protein A to be compared with 
Yi = (ia Viz» Viz)", the coordinate vector of the ith atom selected from protein B. Assume that X and Y 
have been translated so that their centers of geometry are located at the same position, say, at the origin. 
Then, the structural difference between the two proteins can be measured by using the root-mean-square 
deviation (RMSD) of the structures, RMSD(X, Y) = ming ||. X — Y Q||r/./n, where Q is a3 x 3 rotation 
matrix and QQ’ = I, and || - || is the matrix Frobenius norm. 

The RMSD is basically the smallest average coordinate errors of the structures for all possible rotations 
Q of structure Y to fit structure X. It is called the Procrustes problem for its analogy to the Greek story 
about cutting a person’s legs to fit a fixed-sized iron bed. Note that X and Y may be the coordinate matrices 
for the same (A = B) or different (A # B) proteins and therefore, each pair of corresponding atoms do 
not have to be of the same type (when A 4 B). However, the number of atoms selected to compare must 
be the same from A and B (# rows of X = # rows of Y). 


Facts: 


1. Let Aand B be two matrices. Suppose that A is similar to B, then trace( A) = trace(B). In particular, 
trace(A) = trace(V" AV), for any orthogonal matrix V. 
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2. [GL89] LetC = Y’XandC = UXV’ bethe singular-value decomposition of C. Then, Q = Uv? 
minimizes ||X — Y Q]|r. 


Algorithm 2: Computing the RMSD of Two Protein Structures 


1. Compute the geometric centers of X and Y: 


=, ARR. aes 


velil=(S7_ Ye A)im fat 3. 


2. Translate X and Y to the origin: 
XS Xie, YSY Sig 2. Tee Oyeaas OR 
3. Compute C = Y’X and C = UV". Then, 


Q=UV", RMSD(X,Y) = ||X — YQ\|p//n. 


Examples: 


1. Suppose that X and Y are given as the following. 


—-l1 -l -2 
-1 -1 0 
a1, 2 2 
ie —-1 1 0 
kl) 2 
1 -l 0 
1 1, -=2 
1 1 0 
1 1 
1 -l 
—-1 1 
Y= 


| 
— 
| 
— 
N NNN OOO Oo 
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Then x, = (0,0,—1)7 and y, = (0,0, 1)!. Following the Step 2 in Algorithm 2, X and Yare 


changed to 
al. =k 1 
oo! nd Ge | 
-l1 1 -!l 
a —-1 1 1 
1 —-l -l 
1 -1 1 
1 1 -l 
1 1 1 
1 1 -l 
1 -l -l 
=): -=l 
= =] ty eT 
1 1 1 
1 SSI. 
-1 1 1 
-1 -1 1 
Let C = Y’ X. Then, 
0 —-8 O 
C= |0 0 -8 
8 0 0 


Compute the singular value decomposition of C to obtain C= UXV’, with 


0 1 0 0 1 0 
U=]|]0 0 -1] and V=/]0 O -1 
-1 0 0 -1 0 0 
Then, 
0 -1 0 
Q=UV'=|0 0 -1 
1 0 0 


By calculating || X — Y Qllz / ./n, we obtain RMSD (X, Y) = 0. 

2. RMSD calculation has been widely used in structural computing. A straightforward application is 
for comparing and validating the structures obtained from different (x-ray crystallography, NMR, 
or homology modeling) sources for the same protein [Rho00]. Even from the same source, such as 
NMR, multiple structures are often obtained, and the average RSMD for the pairs of the multiple 
structures has been calculated as an indicator for the consistency and sometimes the flexibility 
of the structures [SNB03]. It has also been an important tool for structural classification, motif 
recognition, and structure prediction, where a large number of different proteins need to be aligned 
and compared [EJT00]. Figure 60.3 gives an example of using RMSD to compare NMR and x-ray 
crystal structures. Three structures of the second domain of the immunoglobulin-binding protein 
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FIGURE 60.3. NMR and x-ray crystal structures of 2IGG: Two NMR structures of 2IGG are superposed to its x-ray 
crystal structure to find out which one is closer to the x-ray crystal structure (dark line). The RMSD values for the two 
NMR structures against the x-ray structure are 1.97 Aand 1.75 A, respectively. (Photo courtesy of Feng Cui.) 


(2IGG) [LDS92] are displayed in the figure. Two of them are NMR structures. They are compared 
using RMSD against the x-ray structure (dark line). 


60.4 The Karle—Hauptman Matrix in X-Ray 
Crystallographic Computing 


X-ray crystallography has been a major experimental tool for protein structure determination and is 
responsible for about 80% of 30,000 protein structures so far determined and deposited in the Protein 
Data Bank [BWFO00]. The structure determination process involves crystallizing the protein, applying x-ray 
to the protein crystal to obtain x-ray diffractions, and using the diffraction data to deduce the electron 
density distribution of the crystal (Figure 60.4). Once the electron density distribution of the crystal is 
known, a 3D structure for the protein can be assigned [Dre94]. 


Definitions: 


Define p: R°—R to be the electron density distribution function for a protein and Fy in complex 
space C to be the structure factor representing the diffraction spot specified by the integral triplet 
H [Dre94]. 


FIGURE 60.4 Example diffraction image and electron density map: The left one is the diffraction image of a 12-atom 
polygon generated by the program in [PNBO1]. The right one is the electron density map of benzene generated by 
Stewart using program DENSITY in MOPAC [Ste02]. 
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A Karle-Hauptman matrix for a set of structure factors {Fy : H = Ho,..., Hy—1} is defined as 


Facts: 


Fur Fu, <1: Fr 


Fy, Fu. ++: Fm 


. [Dre94] The electron density distribution function p can be expanded as a Fourier series with the 


structure factors F;, as the coefficients. In other words, Fy is a Fourier transform of p. 
= = + ryT 
p(r)= Dad Fy exp(—27i Hr), 
Fy= | p(r) exp(2xiH!r)dr. 
RB 


[Bri84] If K is a Karle-Hauptman matrix, then the inverse of K is also a Karle-Hauptman matrix 
and can be formed directly as 


Em Ena En, 
En, En En, 
KT= ; 
Ey, Eu. +: Em 
where 
En, = p\(r) exp(2xi Hj r)dr, f= O15 e505 Meals 
R3 


[GL89] Suppose that we have a linear system Kx = h, where K is ann x n Karle-Hauptman matrix 
and h an n-dimensional complex vector. If a conventional method, such as Gaussian Elimination, 
is used, the solution of the system usually takes O (n°) floating-point operations, which is expensive 
if n is larger than 1000 and if the solution is also required multiple times. 

[Loa92], [WPTO1] Since each element in the inverse matrix can be obtained by doing a Fourier 
transform for the inverse of p and only n distinct elements in the first column are required to form 
the whole matrix, the calculations can be done in O (nlogn) floating-point operations by using the 
Fast Fourier Transform. 

[TML97], [WPT01] The matrix K~! as well as K has only n distinct elements listed repeatedly in 
the columns of the matrix with each column having the elements in the previous column circulated 
by one element from top to the bottom and then bottom to the top. This type of matrix is called the 
circulant matrix. According to the discrete convolution theory, if h is the Fourier transform of f, then 
K~'h can be computed by doing a Fourier transform for p~! - t, where ¢ can be obtained through 
an inverse Fourier transform for h and the product - is applied component-wise. Therefore, the 
whole computation for the solution of Kx = h can be done with at most O (nlogn) floating-point 
operations. 
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Examples: 


1. Leto = [0.1250, 0.1250, 0.5000, 0.1250, 0.1250] bean electron density distribution. Then p~! = [8, 
8, 2, 8, 8], and the Fourier transforms for p and p~! are equal to 


F = [0.2000, —0.0607 + 0.0441i, 0.0232-0.0713i, 0.0232 + 0.0713i, -0.0607-0.0441i] and 
Finy = [6.8000, 0.9708 — 0.7053i, —0.3708 + 1.1413i, —0.3708 — 1.14131, 0.9708 + 0.7053i], 


respectively. And it is not hard to verify that the inverse of the Karle-Hauptman matrix K(F) 
formed by using F is equal to the Karle-Hauptman matrix Kj, (Finy) formed by using Finy. 

2. The Karle-Hauptman matrix is an important matrix in x-ray crystallography computing, named 
after two Nobel Laureates, chemist Jerold Karle and mathematician Herbert Hauptman, who re- 
ceived the Nobel Prize in chemistry in 1985 for their work on the phase problem in x-ray crystal- 
lography [HK53]. The Karle-Hauptman matrix is frequently used for computing the covariance 
of the structure factors [KH52] or the electron density distribution that maximizes the entropy of 
a crystal system [WPTO1]. 


60.5 Calculation of Fast and Slow Modes of Protein Motions 


In a reduced model for protein, a residue is represented by a point, in many cases, the position of the 
backbone atom C, or the sidechain atom Cg in the residue, and a protein is considered as a sequence 
of such points connected with strings [HL92]. If the reduced model of a protein is known, a so-called 
contact map can be constructed to show how the residues in the protein interact with each other. The map 
is represented by a matrix with its i, j-entry equal to —1 if residues i and j are within, say 7A distance, 
and 0 otherwise. The contact matrix can be used to compare different proteins. Similar contact patterns 
often imply structural or functional similarities between proteins [MJ85]. When a protein reaches its 
equilibrium state, the residues in contact can be considered as a set of masses connected with springs. A 
simple energy function can also be defined for the protein using the contact matrix. 


Definitions: 


Suppose that a protein has n residues with n coordinate vectors x, ..., x,. A contact matrix I" for the 
protein in its equilibrium state can be defined such that 


‘ -1, llxs—xjll <7A 
i,j = ey el ee 

0, otherwise [HBE97]. 
Aiiw=— ja Ang i= Len 


A potential energy function E for a protein at its equilibrium state can be defined such that for any 
vector Ax = (Ax;,..., Ax,)" of the displacements of the residues from their equilibrium positions, 


1 , 
E(Ax) = 5 k Ax’ A Ax, 


where k is a spring constant. 
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FIGURE 60.5 Mean-square fluctuations: The fluctuations for protein 2KNT based on the mean-square fluctuations 
calculated with GNM (a) and the B-factors determined by x-ray crystallography (b). The two sets of values show a 
high correlation (0.82) (c). (Photos courtesy of Di Wu.) 


Facts: 


1. [HBE97] Given a potential energy function E, the probability for a protein to have a displacement 
Ax at temperature T should be subject to the Boltzmann distribution, 


pr(Ax) = 5 ep(-E(Ax)/ks T)= 5 exp(-k Ax! A Ax/2kgT), 


where Z is the normalization factor and kg the Boltzmann constant. 
2. [HBE97] Let the singular-value decomposition of I" be given as T = UAU?. Then, the mean- 
square residue fluctuations of a protein at its equilibrium state can be estimated as 


1 n ee 
< Ax;, Ax; > = Al Ax! Ax; exp(—E(Ax)/kgT)dAx = Dae kel U, (EF; Uj, j/k. 
R3n = 


i,jj, 
: hi 


Examples: 


1. The energy model defined above for a protein at its equilibrium state is called the Gaussian Network 
Model. The model can be used to find how the residues in the protein move around their equilibrium 
positions dynamically and in particular, to estimate the so-called mean-square fluctuations for the 
residues <Ax;, Ax;>, i = 1, ..., n. If the mean-square fluctuation is large, the residue is called 
hot, and otherwise, is cold, which often correlates with the experimentally detected average atomic 
fluctuation such as the B-factor in x-ray crystallography [Dre94] and the order parameter in NMR 
[Gun95]. In fact, the Gaussian Network Model is equivalent to the Normal Mode Analysis for 
predicting the mean-squares residue fluctuations of a protein, with the energy function defined for 
the residues instead of the atoms. 

2. Figure 60.5 shows the mean-square fluctuations calculated using the Gaussian Network Model 
for the protein 2KNT and the comparison with the B-factors of the structure determined by x-ray 
crystallography. The two sets of values appear to be highly correlated. Based on the facts stated above, 
the calculation of the mean-squares fluctuations requires only a singular-value decomposition of 
the contact matrix for the protein. 


60.6 Flux Balancing Equation in Metabolic Network Simulation 


A metabolic system is maintained through constant reactions or interactions among a large number 
of biological and chemical compounds called metabolites [Fel97]. The reaction network describes the 
structure of a metabolic system and is key to the study of the metabolic function of the system. Figure 60.6 
shows the reaction network for an example metabolic system of five metabolites given in [SLP00]. 
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FIGURE 60.6 Example metabolic networks: A, B, C, D, E are metabolites; vj, j = 1,..., 6 are internal fluxes; b;, 
j =1,...,4 are external fluxes. Each flux v; corresponds to an internal reaction. 


Definitions: 


Each metabolite has a concentration, which changes constantly. The rate of the change is proportional to 
the amount of the metabolite consumed or produced in all the reactions. 

Let C; be the concentration of metabolite i. Let v; be the chemical flux in reaction j, i.e., the amount 
of metabolites produced in reaction j per mole. Then, 


dC; n 
it — 5 Si,j Vio 


where s;,; is the stoichiometric coefficient of metabolite i in reaction j, and s;,; = +k, if +k moles of 
metabolite i are produced (or consumed) in reaction j. 

Let C = (Cj,...; Cm)? be a vector of concentrations of m metabolites, and v = (v),...,V,)! a vector 
of fluxes of n reactions. Then, the equations can be written in a compact form, 


dC 5 
= Sy, 
dt 
where S = {s,;:i=1,...,mj=1,..., n} is called the stoichiometry matrix, and the equations are 


called the reaction equations [HS96]. 

The fluxes are functions of the concentrations and some other system parameters. Therefore, the above 
reaction equations are nonlinear equations of C. However, when the system reaches its equilibrium, 
dC/dt = Sv = 0, and the vector v becomes constant and is called a solution of the steady-state flux 
equation Sv = 0 [HS96]. 

The steady-state fluxes are important quantities for characterizing metabolic networks. They can be 
obtained by solving the steady-state flux equation Sv = 0. However, since the number of reactions is 
usually larger than the number of metabolites, the solution to the equation is not unique. Also, because 
the internal fluxes are nonnegative, the solution set forms a convex cone, called the steady-state flux cone. 

Usually, a convex cone can be defined in terms of a set of extreme rays such that any vector in the cone 
can be expressed as a nonnegative linear combination of the extreme rays, 


1 
cone(S) = {v= YS WiPis Wi = O}, 


where p = {pi,..., pr} is a set of extreme rays. An extreme ray is a vector that cannot be expressed as a 
nonnegative linear combination of any other vectors in the cone. 

A set of vectors is said to be systematically independent if none of them can be expressed as a nonnegative 
linear combination of others. Since the extreme rays can be used to express all the vectors in a convex cone, 
they are also called the generating vectors of the cone. For metabolic networks, they are called the extreme 
pathways [PPP02]. 
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Facts: 


1. [PPP02] A convex flux cone has a set of systematically independent generating vectors or extreme 
pathways. They are unique up to positive scalar multiplications. If the extreme pathways of the 
convex flux cone of a metabolic network are found, all the solutions for the steady-state flux 
equation can be generated by using the extreme pathways. In other words, the extreme pathways 
provide a unique description for the solution space of the steady-state flux equation, and can be 
used to characterize the whole steady-state capacity of the system. 

2. [PPP02] Let P be a matrix with each column corresponding to an extreme pathway of a given 
metabolic network. Let Q be the binary form of P such that Qj; = lif P;,; #0 and Qj; = 0 
otherwise. Then, the diagonal elements of Q Q are equal to the lengths of the extreme pathways, 
while the diagonal elements of QQ? show the numbers of extreme pathways the reactions participate 
in. 


Examples: 


1. Consider the example network in Figure 60.6 and let S be the stoichiometric matrix with 10 columns 
for the internal (v1, ..., v6) as well as external (b,,..., b4) fluxes, 


V1 V2 V3 Ve V5 Ve db, by bz dg 


—1l 0 0 0 0 0 -1 0 0 O|<A 
1 =) wd 0 0 0 0 —-1 0 0o|<B 
S=1] 0 1 -l1 -1l 1 -1 0 Oo; <C. 
0 0 0 1 -l 0 0 0 -1 0O0]}]<D 
0 0 0 0 0 1 0 0 0 -l| <E 


Then, by using an appropriate algorithm (such as the one given in [SLP00]), a matrix of 7 extreme 
pathways of the system can be found as follows: 


V1 V2 V3 V4 V5 Vo Bb, bp b3 bg 


10000 0-1 #1 0 O| <p, 
01 1000 0 0 0 O| < po 
010100 0 j-1 1 OO}; <— ps 
PES 10 7 °C. 0-0 1 0 Sl OT py 
00101 0 0 1 —-1l O} < ps 
00011 0 0 0 0 O|] < pe 
0000121 0 0 -—-1 1] <p, 
where row i corresponds to extreme pathway p;,i=1,..., 7. 


2. By forming the binary form Q for P and computing 


Q 
< 
RQ 
II 

Co OF FF OW 
Cc OF FF NYO 
ee NY NY Fk | 
ie) (an) e aS NO e — 
oF BR RF NO Re eS 
ep NY F-|§ CO KF CO FG 
Pre NY NY KF OO OS 
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and 
10000 0 1 1 0 0 
0 3 121 0102 «1~21 
0 120100 21 1 £=0 
0 1°02 10 0 1 1 =0 
001 1 3 10 1 2 «21 
QO" = , 
0 1 0012 021 21 2 
10000 0 1 1 0 0 
12 121 212121 4 2~«21 
0 1 1 12 2102 3~21 
0 100312 021 21 2 


we obtain, from the diagonal elements of the matrices, the lengths of the pathways, pi: 3, po: 2, p3: 
4, pa: 4, ps: 4, po: 2, p7: 4, and the participations of the reactions in the extreme pathways, v,: 1, 
V2: 3, 73: 2, Vat 2, V5: 3, V6: 2, Dy: 1, bo: 4, b3: 3, bg: 2. The off-diagonal elements of Q™ Q show the 
numbers of common reactions in different extreme pathways. For example, (Q? Q)34 = 2 means 
that p3 and p,4 share two common reactions, and (Q? Q)35 = 1 means that p3 and ps have one 
common reaction. The off-diagonal elements of QQ’ show the numbers of extreme pathways in 
which different reactions participate. For example, (QQ™)2.3 = 1 means that v2 and v3 participate 
in one extreme pathway together, and (QQ™)3,3 = 2 means that v, and by participate in two extreme 
pathways together. 


60.7 Conclusion 


In this chapter, we have reviewed several subjects in biomolecular modeling, where linear algebra has played 
a central role in related computations. The review has focused on simple showcases and demonstrated 
important applications of linear algebra in biomolecular modeling. The subjects discussed are of great 
research interest in computational biology and are related directly to the solutions of many critical but 
challenging computational problems in biology yet to be solved, including the general distance geometry 
problem in NMR, the phase problem in x-ray crystallography, the structural comparison problem in 
comparative modeling, molecular dynamics simulation, and biosystems modeling and optimization, which 
we have not elaborated in detail here, but the interested readers can further explore. 

Linear algebra has also been used to support many basic algebraic calculations required for solving other 
types of mathematical problems in biomolecular modeling, such as the optimization problems in potential 
energy minimization [WS99], the initial value problem in molecular dynamics simulation [BKP88], and 
the boundary value problem in simulation of protein conformational transformation [EMO99]. They are 
usually straightforward or routine linear algebra calculations, so we have not covered them in this chapter, 
but they should be considered as equally important as the applications we have discussed. 
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Sometimes errors occur when data is transmitted over a channel. A binary 0 may be corrupted to a 1, 
or vice versa. Error control coding adds redundancy to a transmitted message allowing for detection 
and correction of errors. For example, if 0 stands for “no” and 1 stands for “yes”, a single bit error will 
completely change a message. If we repeat the message seven times, so 0000000 stands for “no” and 
1111111 stands for “yes”, then it would require at least 4 bit errors to change a message, assuming that 
a message is decoded to either 0000000 or 1111111 according to which bit is in the majority in the 
received string. Thus, this simple code can be used to correct up to 3 errors. This code also detects up to 6 
errors, in the sense that if between 1 and 6 errors occur in transmission, the received string will not be a 
codeword, and the receiver will recognize that one or more errors have occurred. The extra bits provide error 
protection. 

Let F = F, be the finite field with q elements. It is customary in the coding literature to write message 
blocks and vectors of the vector spaces F* and F” as row vectors, and we shall follow that practice in this 
chapter. 


61.1 Basic Concepts 


Definitions: 


A block code of length n over F is a subset C of the vector space F”. Elements of C are codewords. Elements 
of the finite field F form the alphabet. 

Message blocks of length k are strings of k symbols from the alphabet. Message blocks may also be 
identified with vectors in the vector space F*. 

An encoder is a one-to-one mapping 


go: Fk — F" 


whose image is contained in the code C. 
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The process of applying ¢ to a k-digit message block to obtain a codeword is encoding. 

A noisy channel is a channel in which transmission errors may occur. 

A codeword v € C C F" that has been transmitted may be received as y = v + e, where e € F” is an 
error vector. 

Given a received word y = v+e € EF", the receiver estimates the corresponding codeword, thus, 
obtaining ¥ € C. This estimation process is called decoding. If ¥ = v, then correct decoding has 
occurred. 

Decoding may be thought of as a mapping w : F" —> C. The mapping yf is a decoder. Generally a 
decoder y should have the property that if the received word is in fact a codeword, it is decoded to itself. 

The word error rate for a particular decoder is the probability Perr that the decoder outputs a wrong 
codeword. 

The process of encoding, transmission, and decoding is summarized by 


where “n.t.” stands for “noisy transmission.” 

For v,w € F” the Hamming distance, or simply the distance, between v and w is d(v, w) = |{i | vi # 
wil. 

The distance of a code C C F” is defined by d(C) = min{d(v, w) | v, w € C and v $ w}. 

The Hamming weight or weight of v is defined to be wt(v) = d(v, 0). 

The closed ball of radius t about the vector v € F” is the set 


B(v,t) = fy € F" | dw,y) < th. 


Nearest-neighbor decoding refers to decoding a received word y to a nearest codeword (with respect 
to Hamming distance). 
A code C C F” is called a perfect code if there exists a nonnegative integer t such that 


1. F” = U,ec B(v, t), and 
2. For distinct v, w € C, B(v,t) MN B(w,t) = ¢. 


That is, F” is the disjoint union of closed t-balls about the codewords. 


Facts: 


1. The minimum number of symbol errors it would take for v to be changed to w is d(v, w). 

2. A nearest-neighbor decoder need not be unique as it is possible that some received words y might 
have two or more codewords at an equal minimum distance. The decoder may decode y to any of 
these nearest codewords. 

3. [Hil86, p. 5] The Hamming distance is a metric on F". 

4. [Hil86, p. 7] Ifthe distance of a code C is d, then C can detect up to d — 1 errors. It can correct up 

d-1 


to [S] errors. 


Examples: 


1. The binary message block 1011 can be identified with the vector (1,0, 1,1) € FS. 

2. The distance between the binary strings 0011 and 1110 is 3 since these two bit strings differ in three 
places. 

3. We may picture nearest-neighbor decoding by taking a closed ball of expanding radius about the 
received word y. As soon as the ball includes a codeword, that codeword is a nearest-neighbor to 
which we may decode y. 
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61.2 Linear Block Codes 


Definitions: 


A linear block code of length n and dimension k over F = F, is a k-dimensional linear subspace, C, of the 
vector space F”. One says C is an [n, k]-code over F. If one wants to include the distance as a parameter, 
one says that C is an [n, k, d]-code over F. A binary linear block code (BLBC) is a linear block code over 
the binary field F, = {0, 1}. 

Two linear codes over F, are equivalent if one can be obtained from the other by steps of the following 
types: (i) permutation of the coordinates of the codewords, and (ii) multiplication of the field elements in 
one coordinate of the code by a nonzero scalar in Fy. 

The rate of a linear [n, k]-code is x 

Let C bea linear [n, k]-code over F. A generator matrix for C is a matrix G € F**" whose k rows form 
a basis for C. A generator matrix for C is also called an encoder for C. 

A generator matrix G for an [n, k]-code is a systematic encoder if G can be partitioned, perhaps after 
a column permutation, as G = [I | A], where A € Fkx("-*) Then u € F* is encoded as uG = [u | uA] 
so that the first k symbols of the codeword uG are the message word u, and the last n — k symbols form 
the parity check symbols. 

Let C be a linear [n, k]-code over F. A parity check matrix for C is a matrix H € F’-*" such that 
C={ve F"|vH? =0e€ FU}, 

Let C be a linear [1,k]-code over F with generator matrix G € F**" and parity check matrix H € 
Fx" Then H itself is the generator matrix of another code, denoted C +. C+ is the dual code of C. 

A linear block code, C, is self-dual ifC = C+. 

Let C be an [n, k]-code with parity check matrix H. For any vector y € F", the syndrome of y is the 
vector yH? € F"-"), 


Facts: 


1. [Hil86, p. 47] There are q* codewords in an [n, k]-code over F = Fy. 
2. The rate, k , ofan [n, k]-code tells what fraction of transmitted symbols carry information. The com- 
plementary fraction, mk , tells how much of the transmission is redundant (for error 
protection). 
3. If G isa generator matrix for the linear [n, k]-code, C, thenC = {uG | ue€ F*}. Thus, the mapping 
y : FX —> F” given by g(u) = uG is an encoder for C. 
4, [Hil86, p. 48] The distance ofa linear code is equal to the minimum weight ofall nonzero codewords. 
5. Every [n, k] linear block code has an (n — k) x n parity check matrix. 
6. Fora given [n, k] linear code C with generator matrix G, an (n — k) x n matrix H isa parity check 
matrix for C if and only if the rows of H are a basis for the left kernel of G’. 
7. If H isa parity check matrix for the linear code C, then C is the left kernel of H io 
A generator matrix of C is a parity check matrix of C+. 
9. [MS77, p. 33] Suppose H € F("—)*" is a parity check matrix for an [n, k]-code C over F. Then the 
distance of C is equal to the smallest number of columns of H for which a linear dependency may 
be found. 
10. [Hil86, p. 70] If G = [J | A] isa systematic encoder for the [n, k]-codeC, then H = [—A™ | Lx] 
is a parity check matrix for C. 

11. [MS77, p. 22]Shannon’s Coding Theorem for the Binary Symmetric Channel. Assume that the trans- 
mission channel has a fixed symbol error probability p that the symbol 1 is corrupted to a 0 or vice 
versa. Let 


2 


C :=14+4 plog, p+ (1 — p)log,(1 — p). 
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Then for every € > 0 and every R < C there exists, for any sufficiently large n, a binary linear 
[n, k] code having transmission rate k/n > R and word error probability Perr < €. 


Examples: 


1. The binary repetition code of length 7 is a [7,1] BLBC. A generator matrix is 


G/U Ra Vd Al, 


This code consists of two codewords, namely 0000000 and 1111111. The rate of this code is i. For 
every 7 bits transmitted, there is only 1 bit of information, with the remaining 6 redundant bits 
providing error protection. The distance of this code is 7. A parity check matrix for this code is the 
6 x 7 matrix 


A= [sex | Tel: 


2. Binary even weight codes. The binary even weight code of length n is a [n, n — 1]-code that consists 
of all even weight words in F5. A generator matrix is given by 


C= es. ||| tesa 


This code detects any single error, but cannot correct any errors. A binary even weight code may 
also be constructed by taking all binary strings of length n — 1 and to each adding an nth parity 
check bit chosen so that the weight of the resulting string is even. The 8-bit ASCII code (American 
Standard Code for Information Interchange) is constructed in this manner. 
Note that the binary even weight code of length n and the binary repetition code of length n are 

dual codes. 

3. The ISBN (International Standard Book Number) code is a [10, 9]-code over F, (the finite field 
{0, 1, 2, 3, 4,5, 6, 7, 8, 9, X} where field operations are performed modulo 11. X is the symbol for 10 
in this field). This code may be defined by the parity check matrix 


He | 293 a Soe 7 8 


Thus, a string of 10 digits, vivov3vavsveV7VgVoV10, from Fy, is a valid ISBN if and only if 
year iv; = 0 (mod 11). The ISBN code will detect any single error or any error where two 
digits are interchanged (transposition error). The first nine digits of an ISBN carry information 
about the book (such as publisher). The tenth digit is a check digit. Given the first 9 digits of an 
ISBN, the check digit may be computed by forming the sum ye , iv; and reducing modulo 11. 
The value thus obtained is v9 (if the value is 10, then vip = X). 

Note: Currently the ISBN system does not use X as any of the first 9 digits. However, mathemat- 
ically there is no reason not to use X anywhere in an ISBN. 

4. The binary linear [8, 4] code with generator matrix 


1 
0 
1 
0 


oF FE Oo 
ese Fe Oo FO 


0 
0 
0 
1 


a > Ee a | 
Se Se eS eS 


0 
1 
0 
0 


a 


is a self-dual code (note that over the binary field, GG! = 0). This code is called the extended 
binary [8, 4, 4] code. All pairs of different codewords are at distance 4 or 8. 


Coding Theory 61-5 


61.3 Main Linear Coding Problem and Distance Bounds 


Definitions: 


Let A, (n, d) be the largest possible number so that there exists a block code C € K of distance d = d(C) 
and cardinality |C| = A, (n, d). The Main Coding Theory Problem asks for the determination of A, (n, d) 
for different values of n, d,q. 

Similarly, let B,(n,d) be the largest possible number so that there exists a linear [n, k] block code C 
of distance d = d(C) and cardinality |C| = B,(n,d). The Main Linear Coding Problem asks for the 
determination of B,(n, d) for different values of n, d, q. 

An [n, k, d] linear code is maximum distance separable (MDS) if d = n—k+ 1. 


Facts: 


1. Finding a value of B,(n, d) is equivalent to finding the largest dimension, k, for which there is an 
[n, k, d]-code over F,. For such maximal k, we have B,(n,d) = qk. 

. B,(n,d) < Ag(n,d) for all n,q,d. 

. [Rom92, p. 170] Aj(n, 1) = By(n,1) = q”. 

. By(n,2) = q”"—|. Such a code is achieved by taking C to be a code with parity check matrix J ,y. 

. [Rom92, p. 170] Aj(n,n) = By (n,n) = q. 

. [Rom92, p. 170] For n > 2, Ag(n,d) < qAq(n — 1,d). 

. [Hil86, Theorem 14.5] B,(n,3) = q”", where r is the unique integer such that 


ND & W NH 


(q’'—1)/(q—-1) <n < (q’'—D/(q—-D. 


8. [Rom92, Theorem 4.5.7] The sphere-packing bound. Suppose C is a block code of length n over F, 
with distance d. Let tf = || . Then 


Ay (nd) (14 (") —1)+ (3) « Spe: é (q- v) <q". 


9. [Hil86, pp. 20-21] The code C is perfect if and only if equality holds in the sphere-packing bound. 
10. [Rom92, Theorem 4.5.7] Suppose C is a block code of length n over F, with distance d. Then 


Ain) aq". 
11. The Singleton bound for linear block codes. For any [n, k, d] linear code over F,, we have 
d<n—k+1. 


12. An (n —k) x n matrix H with entries in F is the parity check matrix of an MDS code if and only 
if any n — k columns of H are linearly independent. Equivalently, H has the property that any full 
size minor of H is invertible. 

13. [MS77, pp. 546-547] The Griesmer bound. Let N(k, d) be the length of the shortest binary linear 
code of dimension k and distance d. Then one has 


k-1 d 
N(k,d ld, 
a> > [F) 
i=0 
The right-hand side in the above inequality is, of course, always at least d + k — 1. 


14. [Rom92, Theorem 4.5.16] The Plotkin bound. If d > eo then 


dq 
A, (n, d) < ——+—_. 
q(n )S anon 
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15. [Hil86, pp. 91-92] The Gilbert Varshamov bound. Assume q is a prime power. For any n and d, if k 
is the largest integer satisfying 


then A,(n,d) > qk. 


Examples: 
1. Consider the binary [7,4] code C defined by the parity check matrix 


0001111 
H={01100i41i1 
103103101 


This code has 2* = 16 elements. Since any two columns of H are linearly independent, but the first 
three columns are dependent, it follows that d(C) = 3. So this code meets the sphere packing bound 
with t = 1. Observe that reading downward, the 7 columns of H form the numbers 1 through 7 in 
binary. This leads to a very nice nearest-neighbor decoding scheme that corrects any single error. 
If the received word is y, the syndrome s = yH" consists of 3 bits. Interpret these 3 bits as a binary 
number, x. If x is 0, the received word is a codeword and we assume that no errors have occurred. 
If x is not 0, it is one of the numbers from 1 to 7. In y we change the bit in position x. This will give 
us a codeword (the unique codeword) at a distance of 1 from y. We presume that this codeword 
is the intended transmission. For instance, if we receive the vector y = 1100100, we compute the 
syndrome s = yH? = 110. Interpreting 110 as a binary number, we obtain x = 6. We change the 
sixth bit of y and obtain the codeword 1100110. This codeword is at a Hamming distance 1 from 
y, and we assume it was the intended transmission. 


2. Assume @1,...,Q@, are pairwise different nonzero elements of a finite field F. Let 
Oy 05) ay 
2 2 2 
ay i) an 
A: 
n—k n—k n—k 
Oy 3) a, 


Then any full-size minor is the determinant of a Vandermonde matrix and, hence, nonzero. So H 
represents the parity check matrix of an MDS code. 


61.4 Important Classes of Linear Codes 1 


Definitions: 
Let F = Fy. 

Let r be a positive integer. Let P(IF’) be the set of all 1-dimensional subspaces of F”. Choose a basis 
vector for each element of P(F’ ). Let H be a matrix whose columns are these basis vectors. The matrix H 
is the parity check matrix of a Hamming code, C. 

A cyclic code is a linear code C for which every cyclic shift of a codeword is again a codeword. 

Let F[x] be the polynomial ring in one indeterminate over F. Let (x" — 1) be the ideal generated by 
x" — 1. The quotient ring F[x]/(x” — 1) is denoted R,,. The polynomials of R,, are identified with vectors 
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1 


i) = 
Ag + ayx + anxX* + +++ dy_1x" ~t > [ag a, ao +++ Ay_y].- 


Suppose k < n are positive integers andr = n — k. A polynomial g(x) = ay + a,x + ax” +++» +a,x" 
dividing x" — 1 € F[x] is a generator polynomial of a cyclic [n, k]-code, C. The polynomial h(x) = 
bo + bix + box? +--+ + bex* chosen so that g(x)h(x) = x" — 1 is the parity check polynomial of C. 


Facts: 


1. The ring R,, forms an n-dimensional vector space over F. The monomials {1, x, x7,...,x"—} form 
a basis. It further has an algebra structure over F (i.e., there is a well-defined multiplication). In fact, 
polynomials in R,, are multiplied as usual, but x” = 1 in R,, so exponents are reduced modulo n. 

2. The cardinality of P(F") ism = (q" — 1)/(q — 1). Thus, the parity check matrix, H, of a Hamming 
code is of size r x n. The corresponding Hamming code C has q* elements where k = n — r. 
Furthermore, the distance d(C) = 3. The Hamming code reaches the sphere packing bound and 
hence is a perfect code in all cases. 

3. [Hil86, pp. 147-148] For cyclic codes, R, has the structure of a principal ideal ring (any ideal is 
generated by a single polynomial). Thus, any cyclic code has a generator polynomial g(x) where 
g(x) is a divisor of x” — 1. Moreover, every cyclic code has a parity check polynomial, namely 
h(x) = (x" — 1)/g(x). 

4, [Hil86, Chap. 12] IfC isa cyclic code with generator polynomial g(x) = ag+a,x+apx?+++++a,x" 
and parity check polynomial h(x) = bo +b)x+b2x7?+---+b .x*, then the corresponding generator 
matrix, G, and parity check matrix, H, of C are given by 


Ag A, ag s+ ay 0 
0 do a, a ++: a O 
G= 0 0 a a a «+: a 
0 O 0 a a a a, 
and 
by ess bp by og 60s 0 
O bk «++ by by bo 
H= 0 0 by bo b, bo 0 
0 O 0 by bo by bo 
Examples: 


1. Forq = 5 andr = 2, we get the [6, 4, 3]- Hamming code over Fs with parity check matrix 


LO. TL yd ad 
A= 
01123 4 
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and generator matrix 


4 4 10 0 0 

43 01 0 0 
G= ‘ 

42 00 1 0 

4 100 0 1 


2. The binary repetition code of length n isa cyclic code with generator polynomial 1+x+---+x"7!. 
The binary even weight code is a cyclic code with generator polynomial 1 + x. 

3. Over F,, the polynomial x7 — 1 = x7+1=(1+x)1+x+2x3)(14+ x? + x°). Hence, g(x) = 
1+ x + x? is the generator polynomial of a binary [7,4]-cyclic code. The corresponding parity 
check polynomial is h(x) = (1+ x)(1+x7+x3) =1+x+x? +x‘. So, the generator matrix G 
and parity check matrix H for this code are given by 


1 10410 0 0 
1041 110 0 

0 1 10410 (0 
G= and H=]j0 104111 0 

00110 i210 
0 01 012141 

00011 0 1 


Noting that the columns of H represent all the nonzero elements of F3, we see that this code is an 
equivalent version of the Hamming [7, 4]-perfect code. 


61.5 Important Classes of Linear Codes 2 


Definitions: 


The Golay codes [Gol49] are two cyclic codes defined as follows. 

The binary Golay code, G3, is a [23, 12,7] binary cyclic code with generator polynomial g(x) = 
(x! $ x! $ x64 x5 444x241), 

The ternary Golay code, Gj), is an [11,6,5] cyclic code over the ternary field F; with generator 
polynomial g(x) = (x° + 2x7 +x? +4+2x +2). 

Denote by F[x;k — 1] the set of all polynomials of degree at most k — 1 over F. Assume F has size 
|F| > n and let w be a primitive of F (i.e., a is a generator of the cyclic group F*). Define the linear map 
(called the evaluation map) by 


ev: Fl[x;3k-—1] —> Fr" 


f rt > (f(a),...,f(a")). 
The image of this map is a Reed-Solomon code. 


Let F = F, bea fixed finite field and consider the polynomial p(x) = x" — 1 € F[x] and let the root 
w of p(x) bea primitive nth root of unity. Let b, d be positive integers. 


B,(n,d,b,w) := {c(x) € Flx;k — 1] | e(w?) = 0 fori =0,...,d —2} 
is called a Bose-Chadhuri—Hocquenghem (BCH) code having designed distance d. In the special case 


when n = q™ — 1, B,(n,d, b, w) is a primitive BCH code; and if b = 1, B,(n, d, b, w) is a narrow sense 
BCH code. 
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Facts: 
1. Over the binary field F, 


xP4.a (xa xP tx txt xtg x? t Yer tx? tx? tx6 tx 5 txt 1)(x4+ 1). 


Therefore, the binary Golay code, G3, has parity check polynomial h(x) = (x!! + x? +x7+x® + 
xe +x+1)(x+1). 

2. [Hil86, pp. 153-157] The binary Golay code G43 is a [23, 12, 7] code having 4096 elements. G3 is 
a perfect code. 

3. Over the ternary field F3, 


xt = (x? 427 + x? + 2x + 2)(x° + xt + 2x7 + x? + 2)(x +2). 


Therefore, the ternary Golay code, G1, has parity check polynomial h(x) = (x°-+x4+2x3+x?+2) 
(x + 2). 

4, [Hil86, pp. 157-159] The Golay code Gj is a ternary [11, 6,5] code having 729 elements. G1, is a 
perfect code. 

5. [Hil86, Theorem 9.5] Here is a complete catalog of perfect linear codes up to equivalence: 


(a) The trivial perfect codes: All of F”, and the binary repetition codes of odd length. 
(b) The Hamming codes. 
(c) The Golay codes G23 and Gy. 
6. Regarding Reed-Solomon codes, the linear map ev is one-to-one. Hence, the defined Reed— 


Solomon code is an [n,k] code. By the fundamental theorem of algebra the minimum weight 
of a nonzero code word is n — k + 1. It follows that a Reed-Solomon code is MDS. With regard to 


the natural basis 1, x,...,x*~! of F[x;k — 1], this code has a generator matrix: 
1 1 Sas 1 
a a? pe a” 
G:= 
ako! g 2k) gtkK-D) 


7. Regarding BCH codes, it is immediate from the definition that B,(n,d, b, w) has a parity check 
matrix of the form 


1 w? GaN wln-Db 

l woth 2 pw (!-o+1) 
As 

1 word-2 wu. f p(n—l(b+d—2) 


We would like to stress that B,(n,d,b,w) is the F, kernel of H and that the entries of H are in 
general not in the base field F,. Assume w ¢€ F,:, where F,; is an extension field of F,. The F,: 
kernel of H is a Reed-Solomon code of distance d. As a consequence, we have that B, (n, d, b, w) 
has distance at least d. 

8. [Rom92, Chap. 8] The BCH code B, (n, d, b, w) is a cyclic code. Fori = 0,...,d — 2, let mj(x) € 
F, [x] be the minimal polynomial of w’*’ and let 


g(x) = Iem{m;(x) |i = 0,...,d — 2} € F, [x]. 
Then g(x) is a generator of the cyclic code B,(n, d, b, w) and its dimension is 


dim B,(n, d,b,w) = n — deg g(x). 
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9. 


Handbook of Linear Algebra 


[Mas69], [FWRT94], [Sud97], [KV03] Reed-Solomon codes and BCH codes can be efficiently 
decoded up to half the designed distance using the Berlekamp—Massey algorithm [Mas69]. Exten- 
sions for this algorithm exist for algebraic geometric codes [FWRT94]. These algorithms have the 
major drawback that no soft information can be processed. This means that the decoder needs a 
vector in F” as an input and it is not possible to process real numbers as they naturally occur in 
wireless communication. In 1997 Sudan [Sud97] achieved a major breakthrough when he came up 
with a polynomial time algorithm for list decoding of Reed-Solomon codes. Since that time the 
technique has been vastly improved and generalized and a broad overview is given in [Gur04] and 
[KV03]. 


Examples: 


I 


Algebraic Geometric Codes. In 1977, V.D. Goppa had the idea to vastly generalize the construction 
techniques of Reed-Solomon codes and BCH codes. Goppa codes became famous in 1982 when 
Tsfasman, Vladut, and Zink [TVZ82] constructed an infinite family of codes that exceeds the 
Gilbert Varshamov bound. (For further details, the reader is referred to the extensive literature on 
this subject [Gop81], [Gop88], [LG88], [TV91], and [Wal00]. 

Consider the projective plane P% and let V C Pj be a smooth curve of degree 6. This means 
there is an irreducible homogeneous polynomial g(x, y, z) € F[x, y,z] of degree 6 with 


V = {(a,B,y) € PE | g(a, B,y) = 0}. 


The smoothness is characterized by the fact that there is no point on the curve where the partial 
derivatives vanish simultaneously. 1(V) := F[x, y,z]/ < g > is called the homogeneous coor- 
dinate ring. As [(V) is a domain, there is a well-defined quotient field k(V), which is called the 
homogeneous function field. (See e.g. [Ful89, p.91].) 

If f € k(V), then f has an associated divisor 


(f) = So orde(f)P, 
PeV 
where ordp(f) ism if f has a pole of order m at P and —mif f has a zero of order m at P. 
Let D = P| +---+ P, bea divisor on V of degree n. Define the vector space 
L(D) :={f © KV)" | (f) + D = 0} U {0}. 
Let G be a divisor on V having support disjoint from D. Consider the linear map: 


ev: L(G) —> F” 
fo (Ff (Pi), + +> FCP n))- 


The image C(D, G) of the linear evaluation map ev is called an algebraic geometric Goppa (AG) 
code. Note that F[x; k — 1] is a linear space of the form L(G) and in this way Reed-Solomon codes 
are AG codes. 

LetC(D, G) bean algebraic geometric Goppa code defined froma curve of degree 0. Ifdeg G < n, 
then 


1 
dim C(D, G) = dim L(G) — dim L(G — D) > degG+ 1 5 1)(@ — 2) 
and for the distance one has the lower bound 
d(C(D,G)) > n— degG. 


Note that $(0 — 1)(O — 2) is the genus of the curve. One way to construct AG codes with good 
distance parameters is, therefore, to construct curves of low genus having many F, rational points. 
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2. Low density parity check codes. A class of linear codes of tremendous practical importance was 
introduced by R.G. Gallager in his dissertation [Gal63]. These binary linear codes were called low 
density parity check (LDPC) codes by the author and are characterized by the property that the 
code can be written as the kernel of a very sparse matrix. For example, Gallager studied codes whose 
parity check matrix had three 1s in every column and six 1s in every row randomly chosen. 

LDPC codes come with the remarkable property that efficient decoding algorithms exist whose 
complexity increases linearly with the code length [RU01]. One of simplest algorithms is the bit 
flipping algorithm. For this, assume that H is a sparse parity check matrix, a code word v was 
sent, and a vector y = v + e was received. The algorithm then checks bit-by-bit if the weight of 
the syndrome vector yH™ increases or decreases if the bit under consideration is flipped. The bit 
flipping algorithm is one of the most simple type of message passing algorithms. 

LDPC codes were later generalized by R. M. Tanner [Tan81] who defined linear codes defined 
through graphs. A good overview to this active area of research can be found in [MRO1]. The 
class of LDPC codes and codes on graphs became prominent again after the discovery of Turbo 
codes [BGT93]. 


61.6 Convolutional Codes 


Aside from linear block codes, the codes most used in engineering practice are convolutional codes. For 
this, consider the situation where a large amount of data has to be encoded by a linear block code. Let 
mo,...,m, € F* bet + 1 blocks of messages to be transmitted and assume G is the generator matrix of 
a linear [n, k] code. The encoding scheme: 


mt+> mG=x, i=0,...,T 
can be compactly written in the form 
m(z) +> m(z)G = x(z), 


where 


m(z) := S- mz’ €F*[z] and x(z):= So xz! € F"[z]. 
i=0 


i=0 


Elias [Eli55] had the original idea to replace in this encoding scheme the generator matrix G withak x n 
matrix G(z) whose entries are elements of the polynomial ring F[z]. 


Definitions: 


Consider the polynomial ring R := F[z]. A submodule C C R” is called a convolutional code. The rank 
of the code C is the rank, k, of C considered as an R-module. C is an [n, k]-convolutional code. The rate 
of C isk/n. 

A generator matrix for the rate k/n convolutional code C is a k x n polynomial matrix G with entries 
in R such that 


C= {mG =x|me RB‘, xe R"}. 


The degree of the convolutional code C with generator matrix G is defined to be the largest degree of any 
k x k minor of G. 
Let x(z) = ee x;z' € R" bea code vector. Define the weight of x(z) by 


wt(x(z)) := s wt(x;). 
i=0 
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The free distance of the convolutional code C is defined as 


Ufee(C) = min{wt(x(z)) | 0 4 x(z) € C}. (61.1) 


Facts: 


1. [Hun74] R is a principal ideal domain and C is, therefore, a free R-module. As such, C has a 
well-defined rank k, and ak x n generator matrix G with entries in R, such that 


C= {mG =x|me R*, xe R"}. 


2. If G, and G2 are both k x n generator matrices of the same convolutional code C, then there exists 
a unimodular matrix U € Gi,(R) such that G) = UG). 

3. The degree of a convolutional code is well-defined (i.e., it does not depend on the particular gener- 
ator matrix chosen). In the literature the degree is sometimes also called the total memory [LC83] 
or the overall constraint length [JZ99] or the complexity [Pir88]. 

4. The free distance of a convolutional code measures the smallest distance between any two different 
code words. 

5. A major design problem in the area of convolutional codes is the construction of [n, k] codes of de- 
gree 5 whose free distance is maximal or near maximal. This question constitutes the generalization 
of the main linear coding problem. 

6. IfC isan [n, k] convolutional code having G(z) as a generator matrix, then there exists an (n—k) xn 
parity check matrix H(z) with entries in R. H(z) is characterized by the property that G(z) H(z)? = 
OKxk- 

7. Convolutional codes of degree zero correspond to linear block codes. 

8. Convolutional codes of small degree can be efficiently decoded using the Viterbi decoding al- 
gorithm [LC83, Chap. 11]. This algorithm has the distinct advantage that soft decision can be 
processed and due to this fact convolutional codes became the codes of choice for many wireless 
applications. 

9. The free distance of an [n, k] convolutional code of degree 5 must satisfy the generalized Singleton 
bound [RS99]: 


deee(C) < (n—8)([7] +1) +841. 


10. Convolutional codes achieving the generalized Singleton bound are called MDS convolutional 
codes. Such codes exist as soon as the field size is large enough [RS99]. 

11. The literature on convolutional codes is not consistent in terms of terminology and definitions. 
Classical articles are [MS67] and [For70]. The articles [McE98] and [Ros01] provide surveys on 
recent developments and further connections. Generalizations to multidimensional convolutional 
codes were considered and the reader will find further reading and references in [GLRW00]. 

In contrast to the theory of linear block codes, there are not so many algebraic construction 
techniques of convolutional codes with a good designed distance and even fewer constructions of 
codes that can be efficiently decoded. The reader will find algebraic constructions of convolutional 
codes in [GLS04], [Jus75], [Pir88], [RSY96], [SGLRO1], and in the bibliography of these articles. 


Examples: 


1. Consider the finite field F;. The generator matrix G(z) = (z,z + 1,z+ 2,z + 3) defines an 
[n,k] = [4,1] convolutional code C of degree 6 = 1 and distance dfree(C) = 7. The generalized 
Singleton bound for these parameters n, k, and 6 is 8 and the defined code is, therefore, not an MDS 
convolutional code. 
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2. [RS99, Ex. 2.13]. Consider the finite field F7 and the generator matrix 


(22+1) (322+1) (522+1) 
G(z) = : 
(z—1) (z—2) (2z — 3) 


Then G(z) defines an [n, k] = [3, 2] convolutional codeC of degree 5 = 3 and distance dfree(C) = 6. 
The generalized Singleton bound for these parameters n, k, and 6 is 6 and the defined code is therefore 
an MDS convolutional code. 
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Modern computer science emerged when the eminent British mathematician Alan Turing invented the 
concept of Turing machine (TM) in 1936. Though very simple and primitive, TM serves as the universal 
model for all known physical computation devices. The principles of quantum mechanics, another revolu- 
tionary scientific discovery of the 20" century, had never been incorporated in the theory of computation 
until the early 1980s. P. Benioff first coined the concept of quantum Turing machine (QTM). Motivated 
by the problem that classical computers cannot simulate quantum systems efficiently, R. Feynman posed 
the quantum computer as the solution. The field of quantum computation was born. 

Quantum computation mainly studies the construction and analysis of quantum algorithms that out- 
perform the classical counterparts. In terms of computability, quantum computers and classical computers 
possess exactly the same computational power. But in terms of computational complexity, which measures 
the efficiency of computation, there are many examples confirming that quantum computers do solve cer- 
tain problems faster. The two most significant ones are Shor’s factorization algorithm and Grover’s search 
algorithm, among other examples such as the Deutsch—Jozsa problem, the Bernstein—Vazirani problem, 
and Simon’s problem. 

Quantum computers share many common features of the classical computers. In a classical computer, 
information is encoded in binary states (for example, 0 denotes the low voltage state and 1 denotes the high 
voltage state), and processed by various logic gates. In a quantum computer, information is represented 
by the states of the microscopic quantum systems, called qubits, and manipulated by various quantum 
gates. A qubit could be a two-level atom in the excited/ground states, a photon with horizontal/vertical 
polarizations, or a spin-+ particle with up/down spins. The state of a qubit can be controlled via physical 
devices such as laser and microwave. The distinctions between quantum and classical computers originate 
from the special characteristic of quantum mechanics. In contrast to a classical system, a quantum system 
can exist in different states at the same time, an interesting phenomenon called superposition. Superposition 
enables quantum computers to process data in parallel. That is why a quantum computer can solve certain 
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problems faster than a classical computer. But, when we measure a quantum system, it randomly collapses 
to one of the basis states. This indeterministic nature makes the design of efficient quantum algorithms 
highly nontrivial. Another distinctive feature of the quantum computer is that the operations performed 
by quantum gates must be unitary. This is the natural consequence of the unobserved quantum systems 
evolving according to the Schrodinger equation. 

Throughout this chapter, we will use Dirac’s bra-ket notation (see Section 59.4 for more information). 
In quantum mechanics, the state of a quantum system is described by a unit vector in a complex Hilbert 
space (a complete inner product vector space). Under Dirac’s bra-ket notation, we use |y) (“ket”) to 
denote a vector in the Hilbert space and (w| (“bra”) for its dual. The inner product of two vectors |) 
and |¢) is denoted (|p). We also use |yr)|@) and |y@) interchangeably with the notation for the tensor 
product |W) ® Id). 


62.1 Basic Concepts 


Definitions: 


A (classical) Turing machine (TM) is an abstract computing device consisting ofa finite control, a two-way 
infinite tape, and a read/write head that moves to the left or right on the tape. It can be described by a 
6-tuple (Q, A, B,5,qo0,4a), where Q is a finite set of control states, A a finite alphabet, B € A the blank 
symbol, qo, da € Q the initial and accepting states, and 6 the transition function 


6:QxA>Qx Ax {L,R}. 


L and R stand for moving left and right, respectively. 

A quantum Turing machine (QTM) is an abstract computing device consisting of a finite control, a 
two-way infinite tape, and a read/write head that moves to the left or right of the tape. It can be described 
by a 6-tuple (Q, A, B,5,q0; qa), where Q is a finite set of control states, A a finite alphabet, B € A the 
blank symbol, qo, qa € Q the initial and accepting states, and 6 the transition amplitude function 


6:QxAxQxAx{L,R}>C. 


The transition amplitude function satisfies 


DS |8(q1, 41, 42,42, d)|* = 1, 


(q2,a2,d)EQx AX{L,R} 


for any (q,a,) € Q x A. L and R stand for moving left and right, respectively. 

A quantum bit (qubit) is a two-level quantum system, modeled by the two-dimensional Hilbert space 
Hh, with basis {|0), |1)}. For example, for a spin-+ particle, |0) and |1) denote the spin-down and spin-up 
states, respectively. They can be mapped to the standard basis for H) as |0) = [1 0] Tand|1)=[0 1]. 

A quantum register of length n is an ordered system of n qubits, modeled by the 2”-dimensional 
Hilbert space Hy» = H) ® H, ®...® H) with basis {|00...00),|00...01),]00...10),|00...11),..., 
|11...11)}. The basis states are ordered in lexicographic order. We may also write each basis state as |i) 
for 0 <i < 2", interpreting the n-bit string of 0s and 1s as the binary representation of i. 

An one-bit quantum gate is a unitary map U : H) > Hh. 

An n-bit quantum gate is a unitary map U : Hp ® HF) @...® Hy) > H,®@ MH) ®...®@ Mh. 

A quantum circuit on 1 bits is a unitary map on H», which can be represented by a concatenation of 
a finite number of quantum gates. 
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Facts: 
1: 


2. 
3. 


[BV93] Any function which is computable by a TM is computable by a QTM. 

[Fey82] Any function that is computable by a QTM is computable by a TM. 

[NC00, pp. 13] A general state ofa qubit is a unit vector a|0) +b|1), wherea,b € Cand |a|?+|b|? = 
1. a and b are the probability amplitudes of |0) and |1), respectively. A measurement of a qubit 
yields either |0) or |1), with probability |a|? or |b|?, respectively. 

[BB02] A general state of n-bit quantum register is a unit vector 


11...1 


So vel), 


x=00...0 


where w, € C and oe 6 |w.|? = 1. Wy is the probability amplitude of |x). A measurement of 
a quantum register yields |x) € {|00...0),|00...1),...,|11...1)}, with probability |y,|?. 


Examples: 


This section contains a list of quantum gates that are frequently used. We provide the description of 
their effects on the basis states, matrix representations, and circuit diagrams. In the circuit diagrams, the 
horizontal lines stand for the qubits. When there is no gate on the line, no operation is done, which can 
be interpreted as the identity operation. When the diagram of a gate shows up on a horizontal line/lines, 
the corresponding gate operation is applied to the qubit/qubits, with the input coming from the left and 
the output (result) going out to the right. The entire circuit diagram is read from left to right. 


1. 


2: 


3. 


NOT gate Ao: Ao|0) = |1), Ao|1) = |0), or Ay = i Al 


EIS 
Sy, 


The Walsh-Hadamard gate H: H|0) = = (10) + |1)), H]1) = 4((0) —|1)), or 


ec tlinli 4 
li Aa 


The x-rotation gate Xg: X»|0) = cos $10) —isin $/1), Xo|1) = —i sin $10) + cos $1), or 
er cos $ —i sin g 
° | =i sin & cos £} 
2 2 
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4. The y-rotation gate Yg: Yp|0) = cos £|0) + sin $11), Yg|1) = — sin $|0) + cos 2/1), or 


cos’ —sin % 
2 2 

Yo = 6 6 ’ 
sin a cos 2 


5. The z-rotation gate Zy: Zo|0) = e7#9/?|0), Z|1) = e?9/?|1), or 


e 19/2 0 
Zo = 0 ei9/2| 


Z6 


6. Controlled-NOT gate A;: A;|00) = |00), A;|01) = |01), Ai|10) = |11), Ai|11) = |10), or, 


0 0 
0 0 
A, = 
0 1 
1 0 


0 
1 
0 
0 


oo OF 


The first qubit acts as the control bit; the operation on the second qubit is controlled by it. If the 
control bit is |0), no operation is done on the second qubit. If the control bit is |1), the NOT gate 
is applied to the second qubit. There is no change on the control bit in either case. In the diagram 
for the Controlled-NOT gate, a black dot denotes the control bit and a vertical line signifies the 
control action. ® 


ae 
Ie, 


7. Two-bit Controlled-U gate A; (U), where U is any arbitrary one-bit unitary transform: A,(U)|00) = 
|00), A,(U)|O1) = |O1), Ay(U)|10) = |1)U]0), Ay(U)|11) = |1)U]1), or, 


1 0 
0 1 0 0 
Ai(U) = }0 0 
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The first qubit acts as the control bit; the operation on the second qubit is controlled by it. If the 
control bit is |0), no operation is done on the second qubit. If the control bit is |1), the one-bit 
gate U is applied to the second qubit. There is no change on the control bit in either case. In the 
diagram for the Controlled-U gate, a black dot denotes the control bit and a vertical line signifies 
the control action. 


U 


8. Function evaluation operator Us : Hx @ Hyn — Hy ® Hom, where f : {0,1}” > {0,1}”, 
|x) € Hy», and |y) € Hom, is given by 


Ix)|y) > Ix)ly + f(x)mod2”). 


Two special cases of this operator will be used in the algorithms discussed later. 


* m= 1, Uf is given by: |x)|y) > |x)|y @ f(x)), where @ is the addition mod 2. 


Ix) ° U; ° Ix) 


ly) ly ® f (x)) 
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* y = 0, Uf is given by: |x)|0) > |x)| f (x)) 


Ix) Ix) 


|0) If (x)) 


9. Quantum Fourier transform (QFT) (n-bit): 


2"-1 
1 2nixy/2" 
In} > 5 ee ly), 
y=0 
where x € {0,1,...,2” — 1}. This operator is a crucial building block of Shor’s Factorization 


Algorithm. The following example manifests the power of QFT. 

Example: Define a function f : {0,1,2,3} > {0, =} by fM= fB= z and f(0)= f(2)= 
This function has period 2. Consider a 2-bit quantum system with state 5 |1) + |3)), where the 
probability amplitudes of the basis states |0), |1), |2), and |3) are specified by the function values 
of f at 0, 1, 2, and 3. Apply QFT to this state. 


+ rile ) + [3)) 


fice Al 
> Sll0) + it) — [2) — i13)) + = (00) — lt) — [2) + 413)) 


Measurement of the result yields 2, the period of f, with probability i. Thus, QFT provides a tool 
for period finding. 

In the diagram for QFT, X,—1Xn—2 .. .X2%1Xp and Yp_1Vn—2--- Y2¥1 Yo are the binary representa- 
tions of x and y, respectively. 
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Xo e- ° H}— Yo 


x ve I H Ry % 


Xo *** 4H OR Roe Yo 
1 0 
Rg= Ha Ak | 
0 ein / 2? 


62.2 Universal Quantum Gates 


Definitions: 
A 1-bit gate A is special if det(A) = 1. 

A 2-bit gate V is primitive if V is decomposable, i.e., there exist 1-bit gates S and T such that V|xy) = 
S|x) ® Tly), or Vixy) = S|y) ® T|x), for any state |xy). 

A 2-bit gate V is imprimitive if it is not primitive. 

A collection of quantum gates G is universal if, for each n € N, every n-bit quantum gate can be 
approximated with arbitrary accuracy by a circuit consisting of quantum gates in G. 

A collection of quantum gates G is exactly universal if, for each n € N, every n-bit quantum gate can 
be obtained exactly by a circuit consisting of quantum gates in G. 


Facts: 
The following facts can be found in [BBO02]. 


1. The collection of all 1-bit gates and any imprimitive 2-bit gate is universal. 

2. The collection of all 1-bit gates and any imprimitive 2-bit gate is exactly universal. 

3. The collection of all special 1-bit gates and any imprimitive 2-bit gate V with det(V) not being a 
root of unity is universal. 


Examples: 


1. The Xg, Yo, and Zg gates are special. 
2. The NOT gate and Walsh-Hadamard gate are not special. 
1 0 1 0 
3. The 2-bit gate V = % Y 
v211 0 —-1 0 
0 1 0 -l 


is primitive, V|xy) = H|x) ® |y). 
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4. The Controlled-NOT gate is imprimitive. 
5. The collection of all 1-bit gates and Controlled-NOT gate is universal, and exactly universal. 


6. The collection ofall X9, Ys,and Zs gates and Controlled-phase gate A 1(Qg), where Qg = F s| 


and e’® is not a root of unity, is universal. 


62.3 Deutsch’s Problem 


Definitions: 


A Boolean function is a function with codomain {0, 1}. 
A Boolean function f : {0,1} — {0,1} is constant if f(0) = f(1). 
A Boolean function f : {0,1} — {0,1} is balanced if f(0) 4 f(1). 


Problem: Deutsch 


Given a Boolean function f : {0,1} — {0,1}, determine whether it is constant or balanced. 


Algorithm: Deutsch 


1. Prepare a two-bit quantum register and initialize it to the state |0)|1). 
2. Apply the Walsh-Hadamard transform to both qubits. 
3. Apply the function evaluation operator Uy 


Ur = |x)ly) > Ix)ly ® f(x). 


4. Apply the Walsh-Hadamard transform to the first qubit. 
5. Measure the first qubit. If the outcome is |0), f is constant. If the outcome is |1), f is balanced. 


Circuit Diagram: Deutsch 


0.) — H U; H 


a 


Facts: 
The following facts can be found in [CEM98]. 


1. With the classical computer, we need two evaluations of f to determine whether f is constant or 
balanced. 

2. With the quantum computer, we need only one evaluation of f to determine whether f is constant 
or balanced. 
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Examples: 


1. Let f(0) = 0 and f(1) = 1. The following sequence of quantum states shows the result of 
computation utilizing Deutsch’s Algorithm. We start from the initial state |0)|1). Then, 


|0)|1) 
, 10) +11) 10) = 11) _ 10) 10) = 1) 1) 0) = 11) 
V2 V2 ee J2 /2 
|0) 0) — 1). |1) [1) — [0) 
> + 
V2 J2 V2 J2 
[0) 0) — 1) |1) 0) —|1) J) — |1) J0) — |1) 
gf xf? We gf RD ahd 
|0) — |1) 


J2 


=" [1) 


The outcome of measuring the first qubit is |1), hence, f is balanced. 
2. Let f(0) = 0 and f(1) = 0. The following sequence of quantum states shows the result of 
computation utilizing Deutsch’s Algorithm. We start from the initial state |0)|1). Then, 


|0)|1) 
: |0) + |1) |0) — |1) = |0) |0)—|1) — [1) $0) — ]1) 
J2 J2 V2 J2 J2 J/2 
10) |0)—|1) | |1) JO) —|1) 0) + ]1) 0) — ]1) 


v2 V2 V2 V2 A RB 
10) — |) 


a 


— |0) 


The outcome of measuring the first qubit is |0), hence, f is constant. 


62.4 Deutsch-Jozsa Problem 


Definitions: 


A Boolean function f : {0,1}” — {0,1} is constant if f(x) =c,c € {0, 1}, for all x € {0, 1}”. 
A Boolean function f : {0,1}” — {0, 1} is balanced if the function value is 1 (or 0) for exactly half of 
the input values, i.e., card (Gs ({1})) = card (7 ({0})). 


Problem: Deutsch—Jozsa 


Given a Boolean function f : {0,1}" — {0,1}, which is either constant or balanced, determine whether it 
is constant or balanced. 
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Algorithm: Deutsch—Jozsa 


1. Prepare an (n + 1)-bit quantum register and initialize it to the state (|0))"|1). 
2. Apply the Walsh-Hadamard transform to all the qubits. 
3. Apply the function evaluation operator Uy: 


Ur = |x)ly) > Ix)ly ® f(x). 


4. Apply the Walsh-Hadamard transform to the first n qubits. 
5. Measure the first n qubit. If the outcome is |00...0), f is constant. If the outcome is not 
|00...0), f is balanced. 


Circuit Diagram: Deutsch-Jozsa 


10) H HE 


11) H 


D 


Facts: 
The following facts can be found in [CEM98]. 


1. With the classical computer, we need at least 2”~! + 1 evaluations of f to determine with certainty 
whether f is constant or balanced. 

2. With the quantum computer, we need only one evaluation of f to determine with certainty whether 
f is constant or balanced. 


Examples: 


1. Letn = 2and f(00) = f(01) = f(10) = f(11) = 1. The following sequence of quantum states 
shows the result of computation utilizing Deutsch—Jozsa’s Algorithm. We start from the initial state 
|00)|1). Then, 
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|00)|1) 
NO EN IO) VIO) SE = 100) cell) 1) Sea) i) 
ane 2 a 


100) 10) = 11) , 101) 10) = 11), 110) 10) = 11) 111) 10) = 11) 

2 V2 2 V2 2 V2 2 V2 
J00) |1) = 10) , 101) 11) = 10) | 110) 11) = Jo) , 111) 11) = 10) 

2 2 2 2 2 V2 2 2 
|00) |0) —|1)  JO1) 0) — 1) |20) JO) — 1) J) JO) — J) 
2 2 2. fd 2 wp 2 = She 
|00) — |01) — |10) — |11) |) — |1) |0) + |1) 10) + 11) 0) — |1) 
- 2 VE a /2 /2 
[0) — |1) 

a 


— —|00) 


The outcome of measuring the first 2 qubit is |00), hence, f is constant. 

2. Letn = 2, f(00) = f(01) = 0, and f(10) = f(11) = 1. The following sequence of quantum 
states shows the result of computation utilizing Deutsch—Jozsa’s Algorithm. We start from the initial 
state |00)|1). Then, 


|00)|1) 
NO) TP LO) TD 10) A): 100) LOT) + 110) 1) 10) 1) 
v2 V2 V2 J2 


2 

00) JO) —|1) — [O1) 0) —|1) — |10) |0) — 

a iG 2. fi a fh are. 

00) JO) —|1) | [O1) |0) —|1) — |10) |1) — 0) < |11) |1) — |0) 

2 2 2 2 2 2 ao 
J/2 
J2 


1) 2 |11) 10) — |1) 


00) JO) —|1) | [O1) 0) —|1) — [10) 0) —]1) — | 11) 10) — |1) 

a 5) 2 ff 2 2 2 

_ 00) + ]O1) — |10) — |11) |0)—|1) — [O) — |1) JO) + |1) 10) — |1) 
J2 /2 


2 V2 
10) — |1) 


J2 


— |10) 


The outcome of measuring the first 2 qubit is |10), hence, f is balanced. 


62.5 Bernstein-—Vazirani Problem 


Definitions: 


Let x = x1x2...x, and y = y, yz... y, be two n-bit strings from {0, 1}”. The dot product x - y is the mod 
2 sum of the bitwise products: 


XY =X OxY2 ®... O XnYn- 


Problem: Bernstein—Vazirani 
Given a Boolean function f, : {0,1}” — {0,1} defined by f,(x) = a- x, where a is an unknown n-bit 
string in {0, 1}”, determine the value of a. 
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Algorithm: Bernstein—Vazirani 


1. Prepare an (n + 1)-bit quantum register and initialize it to the state (|0))"|1). 
2. Apply the Walsh-Hadamard transform to all the qubits. 
3. Apply the function evaluation operator U;,: 


Uy, = |x)ly) > Ix)ly ® falx)). 


4. Apply the Walsh-Hadamard transform to the first n qubits. 
5. Measure the first n qubits. The outcome is |a). 


Circuit Diagram: Bernstein—-Vazirani 


10) H H |— 


10) H HE 


11) H 


D 


Facts: 
The following facts can be found in [BV93]. 


1. With the classical computer, we need n evaluations of f, to determine the value of a. 
2. With the quantum computer, we need only one evaluation of f, to determine the value of a. 


Examples: 


Let a = 11, a 2-bit string. The following sequence of quantum states shows the result of computation 
utilizing Bernstein—Vazirani’s Algorithm. We start from the initial state |00)|1). Then, 


|00)|1) 
y OT 10) TI) 10) 1): 100) 101) 10) + 11) 10) 1) 
V2 V2 V2 2 V2 


_ 100) 10) 12), 101) 10.) = 1) | 120) 10) = 11), FAD) 10) = 1) 


ge Sg ae a oe 
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00) 10) — |1) , |01) |1) —|0) | [10) |1) — oa |0) — |1) 
Diane 5 BE 3 RO af. a 
00) |0)—|1) — |01) |0)—]1) — 10) |0) — |1)  J11) 10) — |) 
Dip 2 2 2 2 Ue 


00) — |01) — |10) + 11) 10) —|1) _ 10) — |1) 10) — |1) 10) — |) 


2 J2 J2 /2 V2 


The outcome of measuring the first 2 qubits is |11), hence, a = 11. 


62.6 Simon’s Problem 


Definitions: 
A function f : {0,1}” — {0,1}” is 2-1 if for each z € range( f), there are exactly two distinct n-bit strings 
x and y such that f(x) = f(y) =z. 

A function f : {0,1}" — {0, 1}" has a period a if f(x) = f(x ®a), Vx € {0,1}”. 


Problem: Simon 


Given a function f : {0,1}” — {0,1}” which is 2-1 and has period a, determine the period a. 


Algorithm: Simon 


1. Repeat the following procedure for n times. 
(a) Prepare a 2n-bit quantum register and initialize it to the state (|0))"(|0))”. 
(b) Apply the Walsh-Hadamard transform to the first n qubits. 


(c) Apply the function evaluation operator Uy: 
Us : |x)ly) > Ix)ly ® f(x). 
(d) Apply the Walsh-Hadamard transform to the first n qubits. 


(e) Measure the first n qubits. Record the outcome |x). 


2. With the n outcomes x1, x2, ...; Xn, solve the following system of linear equations: 


x):a=0 
xy-a=0 
X,-a=O0. 


The solution a is the period of f. 
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Circuit Diagram: Simon 


10.) —— H H 


10) 


Facts: 
The following facts can be found in [Sim94]. 


1. With the classical computer, we need exponentially many evaluations of f to determine the period 
a. 


2. With the quantum computer, we need O(m) evaluations (on average) of f to determine the period 
a. 


Examples: 


Let f(00) = 01, f(01) = 11, f(10) = 01, and f(11) = 11. The following sequence of quantum states 
shows the result of computation utilizing Simon’s Algorithm. We start from the initial state |00)|00). Then, 


00) |00) 
|0) + [1) 10) + |1) |00) + 01) + |10) + |11) 
53 7} |00) = 5 |00) 
__ 100)]00) |, |01)100) , |10)j00) | |11)100) 
2 2 2 2 
|00)|01)  [O1)|11)  [10)|01) . |11)]11) 
. + 


2 2 2 2 
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__ |00) + |10) |O1) + |11) _ 10) +[1) |0) _ 10) +11) |) 

= so Os > TH) = Fi lO aa wri 
0) +|1 0) —|1 00) + |01 00) — |01 

1) ion) +1) Sy =! ) + | oi) +! i ay. 


The outcome of measuring the first 2 qubits yields either |00) or |01). Suppose that we have run the 
computation above twice and obtained |00) and |01), respectively. We now have a system of linear equations: 


00-a=0 
0l-a=0. 
The solution is a = 10, the period of this function. 


62.7 Grover’s Search Algorithm 


Problem: Grover 
Given an unsorted database with N items, find a target item w. This problem can be formulated using an 
oracle function f : {0,1,...,N— 1} — {0,1}, where 


(OE ‘ ifx Aw, 


1, ifx=w. 


Given such an oracle function, find the w such that f(w) = 1. 


Algorithm: Grover 


Without loss of generality, let N = 2”. 


1. Prepare an (n + 1)-bit quantum register and initialize it to the state (|0))"|1). 
2. Apply the Walsh-Hadamard transform to all the n + 1 qubits. 
3. Repeat the following procedure for about -/N times. Cf. Figure 62.1. 


(a) Apply the function evaluation operator U; (selective sign flipping operator): 
Us : |x)ly) > Ix)ly ® f(x). 


This is equivalent to the unitary operator Z,, = I — 2|w)(w| on the first n qubits. 


(b) Apply unitary operator (inversion about the average operator) Z; = 2|s)(s| — I on the 
first n qubits. Cf. Figure 62.2. 


4. Measure the first n qubits. We obtain the search target with high probability. 


Facts: 
1. [Gro97] With the classical computer, on average, we need O(N) oracle calls (evaluations of f) to 


find the search target. 
2. [Gro97] With the quantum computer, on average, we need O(N) oracle calls to find the search 


target. 
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Circuit Diagrams: Grover 
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lo) — H 
ee eee eae 
lo) — H 
1) H OQ a, 
FIGURE 62.1 Grover’s Algorithm. 
aes e (1) 
ae, ky 
(TN e alle 
Soy Ky 
ae e rials 
Ce ral 
ay amy (ales 
| te 


FIGURE 62.2 Inversion about the average operator Z,. 


3. [BBH98] When N = 4, using Grover’s Algorithm, exactly one oracle call suffices to find the search 


target with certainty. 


Examples: 


Let N = 2? = 4 and Item 3 be the search target, which is encoded by the quantum state |11) (11 is the 
binary representation of 3). The following sequence of quantum states shows the result of computation 
utilizing Grover’s Algorithm. We start from the initial state |00)|1). Then, 


|00)|1) 
NOPE I OAR Oe) WOO) oe LOD MOP EN 10) = 11) 
V2 V2 V2 2 J2 
_ 100) 10) — |1) | 101) 10) — 11) | [20) 10) — ]1) | [11) 10) — |) 
— 2 2 20 V2 20 V2 2 V2 
100) 10) — |1) | 101) 10) —|1) | |10) ]0) —|1) | 11) 11) — ]0) 
2 2 2 eS) 2 2 
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100) 10) = 12) 101) 10) = 11), 110) fo) = 11) __ 11) 10) = 1) 
2 2 2 2 eae i 1 
_ |00) + JO1) + [10) — 11) 0) — 11) 
= - z 

0) — |1) 
a 


=> |11) 


The outcome of measuring the first 2 qubits yields |11), which is the search target w = 3. 


Comments: 


Grover’s Algorithm was discovered by L.K. Grover of Bell Labs in 1996. This algorithm provides a quadratic 
speedup over classical algorithms. Although it is not exponentially fast (as Shor’s Algorithm is), it has a 
wide range of applications. It could be used to accelerate any algorithms related to searching an unsorted 
database, including quantum database search, finding the solution of NP problems, finding the median 
and minimum of a data set, and breaking the Data Encryption Standard (DES) cryptography system. 


62.8 Shor’s Factorization Algorithm 


Integer Factorization Problem: 


Given a composite positive integer N, factor it into the product of its prime factors. 


Algorithm: Shor 


1. Choose a random number a < N; make sure that a and N are coprime. This can be done by 
using a random number generator and Euclidean algorithm on a classical computer. 

2. Find the period T of function fa,n(x) = a* mod N. This step can be further expanded as 
follows: 


(a) Prepare two L-bit quantum registers in initial state 


1 24-1 
—y) m) \0), 
( Ze x=0 
where L is chosen such that N? < 2! < 2N?. 
(b) Apply the function evaluation operator Uy : |x)|0) + |x)| fa,n(x)): 
2-1 2-1 


1 1 
Tt dX Ix}10) > dX. Ix) | fa,nr(x)). 


(c) Apply QFT to the first register: 


1 2k-1 ie! 2-1 : 
Tt Dl fw) > = DO de? 1) | Lf). 
x=0 


y=0 x=0 
(d) Make a measurement on the first register, obtaining y. 


(e) Find T from y via the continued fraction for #7. This step might fail; in that case, repeat 
from 2a. 


3. If T is odd, repeat from Step 1. If T is even and N | (a7/? + 1), repeat from Step 1. If T is even 
and N {(a?/? + 1), compute d = gcd(a™/? — 1, N), which is a nontrivial factor of N. 
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Circuit Diagrams: Shor 


10) H @ — 
10) H ® I— 
10) H r QFT — 


1) 

lo) 

lo) 4 a? gee mee 
lo) 


Facts: 
The following facts can be found in [Sho94]. 
1. The integer factorization problem is classically intractable. The most efficient classical algorithm to 


date, a number field sieve, has a time complexity of O(exp (log N)!/7 (log log N)), 
2. Shor’s quantum factorization algorithm has a time complexity of 


O((log N) log log N log log log N). Hence, it is a polynomial time algorithm. 


Examples: 


Let N = 15. Choose L = 8 such that N? = 225 < 24 < 450 = 2N’. Choose a random integer a = 2, 
which is coprime with 15. Thus, fa,n(x) = 2* mod 15. The following sequence of quantum states shows 
the result of computation utilizing Shor’s Algorithm: 


22-1 


i} 
|0) |0) > 7 4 0 


22-1 


= pe Ix) fax) 


az 


= >, (10)|1) 


1)|2) + |2)|4) + |3)18) 


+ |4)|1) + [5)|2) + |6)|4) + |7)|8) +++» + [28 — 2)|4) + [28 — 1)]8)) 
28-1 28-1 


l 1 ‘ 
& S =< S y x 
> vi af ow” |y)|2* mod 15) 
x=0 y=0 


22-1 28-1 


1 
Bar > ly) ye w*’ |2* mod 15), 
y=0 x=0 
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where w = e2"‘/2", Suppose that the outcome of measuring the first 1 qubits is |56). We can compute the 


continued fraction of to be [0, 4,...]. The second number 4 satisfies 2* = 1 mod 15. So 4 is the period 
of fan. 24/2? — 1 = 3 yields a factor of 15 and 15 = 3 x 5. 


Comments: 


Shor’s Algorithm was discovered by P. Shor of AT&T Labs in 1994. It is the most important breakthrough 
in the research of quantum computation so far. It solves the integer factorization problem, an extremely 
hard problem for classical computers, in polynomial time. The security of the RSA cryptographic system, 
which is widely used nowadays over the Internet, is based on the difficulty of factoring large integers. 
Equipped with a quantum computer, one could easily break the RSA codes with Shor’s Algorithm. 
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Information retrieval is the process of searching within a document collection for information most rel- 
evant to a user’s query. However, the type of document collection significantly affects the methods and 
algorithms used to process queries. In this chapter, we distinguish between two types of document collec- 
tions: traditional and Web collections. Traditional information retrieval is search within small, controlled, 
nonlinked collections (e.g., a collection of medical or legal documents), whereas Web information retrieval 
is search within the world’s largest and linked document collection. In spite of the proliferation of the 
Web, more traditional nonlinked collections still exist, and there is still a place for the older methods of 
information retrieval. 


63.1 The Traditional Vector Space Method 


Today most search systems that deal with traditional document collections use some form of the vector 
space method [SB83] developed by Gerard Salton in the early 1960s. Salton’s method transforms textual 
data into numeric vectors and matrices and employs matrix analysis techniques to discover key features 
and connections in the document collection. 


Definitions: 
For a given collection of documents and for a dictionary of m terms, document i is represented by an 


m x 1 document vector d; whose jth element is the number of times term j appears in document i. 
The term-by-document matrix is the m x n matrix 


A=[did) --- dy] 


whose columns are the document vectors. 


63-1 


63-2 Handbook of Linear Algebra 


Recall is a measure of performance that is defined to be 


# relevant docs retrieved 
0 < Recall = 


# relevant docs in collection — 


Precision is another measure of performance, defined to be 


# relevant docs retrieved 


0 < Precision = - 
# docs retrieved 


Query processing is the act of retrieving documents from the collection that are most related to a user’s 
query, and the query vector q,,x1 is the binary vector defined by 


p= { 1 if term is present in the user’s query, 
. 0 otherwise. 


The relevance of document i to a query q is defined to be 
5; = cos 6; = q'di/lIqllalldill2- 


For a selected tolerance T, the retrieved documents that are returned to the user are the documents for 
which 6; > T. 


Facts: 


1. The term-by-document matrix A is sparse and nonnegative, but otherwise unstructured. 

2. [BBO5] In practice, weighting schemes other than raw frequency counts are used to construct the 
term-by-document matrix because weighted frequencies can improve performance. 

3. [BB05] Query weighting may also be implemented in practice. 

4. The tolerance Tt is usually tuned to the specific nature of the underlying document collection. 

5. Tuning can be accomplished with the technique of relevance feedback, which uses a revised query 
vector such as @ = 6)d, + 53d3 + 57d, where dj, d3, and d; are the documents the user judges 
most relevant to a given query q. 

6. When the columns of A and q are normalized, as they usually are, the vector 6’ =q' A provides 
the complete picture of how well each document in the collection matches the query. 

7. The vector space model is efficient because A is usually very sparse, and q? A can be executed in 
parallel, if necessary. 

8. [BB05] Because of linguistic issues such as polysomes and synonyms, the vector space model 
provides only decent performance on query processing tasks. 

9. The underlying basis for the vector space model is the standard basis e;, e2,. . . , €n, and the orthog- 
onality of this basis can impose an unrealistic independence among terms. 

10. The vector space model is a good starting place, but variations have been developed that provide 
better performance. 


Examples: 


1. Consider a collection of seven documents and nine terms (taken from [BB05]). Terms not in the 
system’s index are ignored. Suppose further that only the titles of each document are used for 
indexing. The indexed terms and titles of documents are shown below. 
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Terms Documents 
T1: Bab(y,ies,y’s) | D1: Infant & Toddler First Aid 
T2: Child(ren’s) D2: Babies and Children’s Room (For Your Home) 


T3: Guide D3: Child Safety at Home 

T4: Health D4: Your Baby’s Health and Safety: From Infant to Toddler 
T5: Home D5: Baby Proofing Basics 

T6: Infant D6: Your Guide to Easy Rust Proofing 

T7: Proofing D7: Beanie Babies Collector’s Guide 

T8: Safety 

T9: Toddler 


The indexed terms are italicized in the titles. Also, the stems [BB05] of the terms for baby (and 
its variants) and child (and its variants) are used to save storage and improve performance. The 
term-by-document matrix for this document collection is 


ae 
=) 


> 

II 
re Oo Or co Coo f& 
oS oO: 6- 2 2. oe = 
SO) tH) Os Os) ses 1S Sr LSS 
et OO et Oo OS 
> > 
Ce 
ee Cte 


For a query on baby health, the query vector is 
q=[1 001000 0 O]?. 
To process the user’s query, the cosines 


q'd; 


6; = cos6; = ———___ 
llall2iIdill2 


are computed. The documents corresponding to the largest elements of 6 are most relevant to the 
user’s query. For our example, 


6 [0 0.40824 0 0.63245 0.5 0 0.5], 


so document vector 4 is scored most relevant to the query on baby health. To calculate the recall 
and precision scores, one needs to be working with a small, well-studied document collection. In 
this example, documents dy, d;, and d; are the three documents in the collection relevant to baby 
health. Consequently, with t = .1, the recall score is 1/3 and the precision is 1/4. 


63.2 Latent Semantic Indexing 


In the 1990s, an improved information retrieval system replaced the vector space model. This system is 
called Latent Semantic Indexing (LSI) [Dum91] and was the product of Susan Dumais, then at Bell Labs. 
LSI simply creates a low rank approximation A, to the term-by-document matrix A from the vector space 
model. 


10. 


. Ais replaced by Ay = a 
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. [Mey00] If the term-by-document matrix A,,.,, has the singular value decomposition A = 


UX VV? = Y%_,ojujv), 0) > op > +++ > 0, > 0, then Ay, is created by truncating this 


expansion after k terms, where k is a user tunable parameter. 


. The recall and precision measures are generally used in conjunction with each other to evaluate 


performance. 
;_, 0iu;v, in the query process so that if q and the columns of A, have 
been normalized, then the angle vector is computed as 67 =q! Ay. 


. The truncated SVD approximation to A is optimal in the sense that of all rank-k matrices, the 


truncated SVD Ay, is the closest to A, and 


A-—Ax|lz = min ||A-B = Jott +07. 
I klle nM Ilr kel ; 


. This rank-k approximation reduces the so-called linguistic noise present in the term-by-document 


matrix and, thus, improves information retrieval performance. 


. [Dum91], [BB05], [BR99], [Ber01], [BDJ99] LSI is known to outperform the vector space model 


in terms of precision and recall. 


. [BR99], [Ber01], [BB05], [BF96], [BDJ99], [BO98], [Blo99], [BRO1], [Dum91], [HBO00], [JLOO], 


[JB00], [LB97], [WB98], [ZBRO1], [ZMS98] LSI and the truncated singular value decomposition 
dominated text mining research in the 1990s. 


. Aserious drawback to LSI is that while it might appear at first glance that A, should save storage 


over the original matrix A, this is often not the case, even when k << r. This is because A is 
generally very sparse, but the singular vectors u; and v/ are almost always completely dense. In 
many cases, Ax requires more (sometimes much more) storage than A itself requires. 


. A significant problem with LSI is the fact that while A is a nonnegative matrix, the singular 


vectors are mixed in sign. This loss of important structure means that the truncated singular value 
decomposition provides no textual or semantic interpretation. Consider a particular document 
vector, say, column 1 of A. The truncated singular value decomposition represents document 1 
as 


Fee eee mere 
i te ke 


so document 1 is a linear combination of the basis vectors u; with the scalar 0,1); being a weight 
that represents the contribution of basis vector i in document 1. What we would really like to 
do is say that basis vector i is mostly concerned with some subset of the terms, but any such 
textual or semantic interpretation is difficult (or impossible) when SVD components are involved. 
Moreover, if there were textual or semantic interpretations, the orthogonality of the singular vectors 
would ensure that there is no overlap of terms in the topics in the basis vectors, which is highly 
unrealistic. 

[Ber01], [ZMS98] It is usually a difficult problem to determine the most appropriate value of k for 
a given dataset because k must be large enough so that A; can capture the essence of the document 
collection, but small enough to address storage and computational issues. Various heuristics have 
been developed to deal with this issue. 
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Examples: 


Ls 


63.3 


Consider again the 9 x 7 term-by-document matrix used in section 63.1. The rank-4 approximation 
to this matrix is 


0.020 1.048 —0.034 0.996 0.975 0.027 0.975 
—0.154 0.883 1.067 0.078 0.027 —0.033 0.027 
—0.012 —0.019 0.013 0.004 0.509 0.990 0.509 
0.395 0.058 0.020 0.756 0.091 —0.087 0.091 
Ag = | —0.154 0.883 1.067 0.078 0.027 —0.033 0.027] . 
0.723 —0.144 0.068 1.152 0.004 —0.012 0.004 
—0.012 —0.019 0.013 0.004 0.509 0.990 0.509 
0.443 0.334 0.810 0.776 —0.074 0.091 —0.074 
0.723 —0.144 0.068 1.152 0.004 —0.012 0.004 


Notice that while A is sparse and nonnegative, A, is dense and mixed in sign. Of course, as k 
increases, Ay looks more and more like A. For a query on baby health, the angle vector is 


6~[.244 .466 —.006 .564 .619 —.030 .619]7. 


Thus, the information retrieval system returns documents ds, dz, d4, do, d), in order from most 
to least relevant. As a result, the recall improves to 2/3, while the precision is 2/5. Adding another 
singular triplet and using the approximation matrix A; does not change the recall or precision 
measures, but does give a slightly different angle vector 


6~[.244 .466 —.006 .564 .535 —.030 .535]’, 


which is better than the A, angle vector because the most relevant document, d4, Your Baby’s Health 
and Safety: From Infant to Toddler, gets the highest score. 


Nonnegative Matrix Factorizations 


The lack of semantic interpretation due to the mixed signs in the singular vectors is a major obstacle in 
using LSI. To circumvent this problem, alternative low rank approximations that maintain the nonnegative 
structure of the original term-by-document matrix have been proposed [LS99], [LS00], [PT94], [PT97]. 


Facts: 
1. 


2. 


If Amn > Ohas rankr, then fora given k < r the goal ofa nonnegative matrix factorization (NMF) 
is to find the nearest rank-k approximation WH to A such that W,,.;% => 0 and Hyy, = 0. Once 
determined, an NMF simply replaces the truncated singular value decomposition in any text mining 
task such as clustering documents, classifying documents, or processing queries on documents. 
An NMF can be formulated as a constrained nonlinear least squares problem by first specifying k 
and then determining 


min ||A— WH; subjectto Waxk >0,  Hkxn = 0. 


The rank of the approximation (i.e., k) becomes the number of topics or clusters in a text mining 
application. 
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3. [LS99] The Lee and Seung algorithm to compute an NMF using MATLAB is as follows. 


Algorithm 1: Lee-Seung NMF 


W = abs(randn(m, k)) % initialize with random dense matrix 
H = abs(randn(k, n)) % initialize with random dense matrix 
fori = 1: maxiter 

H = H.x(W"A)./(W! WH + 107°) % 107° avoids division by 0 

W = W. «(AH!)./(WHH! + 107°) 

end 


4, The objective function || A — WH||7, in the Lee and Seung algorithm tends to tail off within 50 to 
100 iterations. Faster algorithms exist, but the Lee and Seung algorithm is guaranteed to converge 
to a local minimum in a finite number of steps. 

5. [Hoy02], [Hoy04], [SBP04] Other NMF algorithms contain a tunable sparsity parameter that 
produces any desired level of sparseness in W and H. The storage savings of the NMF over the 
truncated SVD are substantial. 

6. Because Aj © oy W;h;;, and because W and H are nonnegative, each column W; can be viewed 
as a topic vector—if wjj,, Wij,,.--5 Wij, are the largest entries in Wj, then terms j;, j2,..., jp dictate 
the topics that W; represents. The entries h;; measure the strength to which topic i appears in basis 
document j, and k is the number of topic vectors that one expects to see in a given set of documents. 

7. The NMEF has some disadvantages. Unlike the SVD, uniqueness and robust computations are 
missing in the NMEF. There is no unique global minimum for the NMF (the defining constrained 
least squares problem is not convex in W and H), so algorithms can only guarantee convergence 
to a local minimum, and many do not even guarantee that. 

8. Not only will different NMF algorithms produce different NMF factors, the same NMF algorithm, 
run with slightly different parameters, can produce very different NMF factors. For example, the 
results can be highly dependent on the initial values. 


Examples: 


1. When the term-by-document matrix of the MEDLINE dataset [Med03] is approximated with an 
NME as described above with k = 10, the charts in Figure 63.1 show the highest weighted terms 
from four representative columns of W. For example, this makes it clear that W, represents heart 
related topics, while W, concerns blood issues, etc. 

When document 5 (column A;) from MEDLINE is expressed as an approximate linear combi- 
nation of W,, W2,..., Wo in order of the size of the entries of Hs, which are 


hos = .1646 > hes = .0103 > h7s = .0045>.-.- > 


we have that 


As © .1646 Wo + .0103 We + .0045 W, + --- 


fatty kidney hormone 
glucose marrow growth 
acids dna hgh 
=.1646] ge, | +.0103 | celts | + -0045 | pituitary | +---- 


insulin nephr. mg 
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Highest Weighted Terms in Basis Vector W., 
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Highest Weighted Terms in Basis Vector W.5 


oxygen 
flow 
pressure 
blood 
cerebral 
hypothermia 
fluid 

venous 
arterial 
perfusion 


Term 
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Highest Weighted Terms in Basis Vector Wg 
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MEDLINE charts. 


Therefore, document 5 is largely about terms contained in topic vector Wg followed by topic vectors 


We and W>. 


2. Consider the same 9 x 7 term-by-document matrix A from the example in Section 63.1. A rank-4 
approximation Ay = Wox4H4,y7 that is produced by the Lee and Seung algorithm is 


Ag 


[0.027 
0 
0.050 
0.360 
0 
0.760 
0.050 
0.445 
| 0.760 


0.888 
0.852 
0.084 
0.172 
0.852 
0.032 
0.084 
0.481 
0.032 


0.196 
1.017 
0.054 
0.073 
1.017 
0.155 
0.054 
0.647 
0.155 


1.081 
0.173 
0.102 
0.729 
0.173 
1.074 
0.102 
0.718 
1.074 


0.881 
0.058 
0.496 
0.179 
0.058 
0.033 
0.496 
0.047 
0.033 


0.233 
0.031 
0.899 
0.029 
0.031 
0.061 
0.899 
0.053 
0.061 
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where 


0.202 0.017 0.160 1.357 


0 0 0.907 0.104 
0.805 0 0 0.008 
0 0.415 0 0.321 
W, = 0 0 0.907 0.104], 
0 0.875 0 0.060 
0.805 0 0 0.008 


0 0.513 0.500 0.085 
0 0.876 0 0.060 


0.062 0.010 0.067 0.119 0.610 1.117 0.610 
0.868 0 0.177 1.175 0 0.070 0 

0 0.878 1.121 0.105 0 0.034 0 

0 0.537 0 0.752 0.559 0 0.559 


Notice that both A and A, are nonnegative, and the sparsity of W and H makes the storage savings 
apparent. The error in this NMF approximation as measured by ||A — WH||7, is 1.56, while the 
error in the best rank-4 approximation from the truncated SVD is 1.42. In other words, the NMF 
approximation is not far from the optimal SVD approximation — this is frequently the case in 
practice in spite of the fact that W and H can vary with the initial conditions. For a query on baby 
health, the angle vector is 


6~[.224 .472 118 .597 .655 .143 .655]’. 


Thus, the information retrieval system that uses the nonnegative matrix factorization gives the same 
ranking as a system that uses the truncated singular value decomposition. However, the factors are 
sparse and nonnegative and can be interpreted. 


63.4 Web Search 


Only a few years ago, users of Web search engines were accustomed to waiting, for what would now seem 
to be an eternity, for search engines to return results to their queries. And when a search engine finally 
responded, the returned list was littered with links to information that was either irrelevant, unimportant, 
or downright useless. Frustration was compounded by the fact that useless links invariably appeared at or 
near the top of the list while useful links were deeply buried. Users had to sift through links a long way 
down in the list to have a chance of finding something satisfying, and being less than satisfied was not 
uncommon. 

The reason for this is that the Web’s information is not structured like information in the organized 
databases and document collections that generations of computer scientists had honed their techniques on. 
The Web is unique in the sense that it is self organized . That is, unlike traditional document collections that 
are accumulated, edited, and categorized by trained specialists, the Web has no standards, no reviewers, 
and no gatekeepers to police content, structure, and format. Information on the Web is volatile and 
heterogeneous — links and data are rapidly created, changed, and removed, and Web information exists 
in multiple formats, languages, and alphabets. And there is a multitude of different purposes for Web data, 
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e.g. some Web pages are designed to inform while others try to sell, cheat, steal, or seduce. In addition, 
the Web’s self organization opens the door for spammers, the nefarious people who want to illicitly 
commandeer your attention to sell or advertise something that you probably do not want. Web spammers 
are continually devising diabolical schemes to trick search engines into listing their (or their client’s) Web 
pages near the top of the list of search results. They had an easy time of it when Web search relied on 
traditional IR methods based on semantic principles. Spammers could create Web pages that contained 
things such as miniscule or hidden text fonts, hidden text with white fonts on a white background, 
and misleading metatag descriptions designed to influence semantic based search engines. Finally, the 
enormous size of the Web, currently containing O(10°) pages, completely overwhelmed traditional IR 
techniques. 

By 1997 it was clear that in nearly all respects the database and IR technology of the past was not 
well suited for Web search, so researchers set out to devise new approaches. Two big ideas emerged 
(almost simultaneously), and each capitalizes on the link structure of the Web to differentiate between 
relevant information and fluff. One approach, HITS (Hypertext Induced Topic Search), was introduced 
by Jon Kleinberg [Kle99], [LM06], and the other, which changed everything, is Google’s PageRank™ that 
was developed by Sergey Brin and Larry Page [BP98], [BPM99], [LM06]. While variations of HITS and 
PageRank followed (e.g., Lempel’s SALSA [LM00], [LM05], [LM06]), the basic idea of PageRank became 
the driving force, so the focus is on this concept. 


Definitions: 


Early in the game, search companies such as Yahoo!® employed students to surf the Web and record key 
information about the pages they visited. This quickly overwhelmed human capability, so today all Web 
search engines use Web Crawlers, which is software that continuously scours the Web for information to 
return to a central repository. 

Web pages found by the robots are temporarily stored in entirety in a page repository. Pages remain 
in the repository until they are sent to an indexing module, where their vital information is stripped to 
create a compressed version of the page. Popular pages that are repeatedly requested by users are stored 
here longer, perhaps indefinitely. 

The indexing module extracts only key words, key phrases, or other vital descriptors, and it creates a 
compressed description of the page that can be “indexed.” Depending on the popularity of a page, the 
uncompressed version is either deleted or returned to the page repository. 

There are three general kinds of indices that contain compressed information for each Web page. The 
content index contains information such as key words or phrases, titles, and anchor text, and this is stored 
in a compressed form using an inverted file structure, which is simply the electronic version of a book 
index, i-e., each morsel of information points to a list of pages containing it. Information regarding the 
hyperlink structure of a page is stored in compressed form in the structure index. The crawler module 
sometimes accesses the structure index to find uncrawled pages. Finally, there are special-purpose indices 
such as an image index and a pdf index. The crawler, page repository, indexing module, and indices, along 
with their corresponding data files, exist and operate independent of users and their queries. 

The query module converts a user’s natural language query into a language that the search engine can 
understand (usually numbers), and consults the various indices in order to answer the query. For example, 
the query module consults the content index and its inverted file to find which pages contain the query 
terms. 

The pertinent pages are the pages that contain query terms. After pertinent pages have been identified, 
the query module passes control to the ranking module. 

The ranking module takes the set of pertinent pages and ranks them according to some criterion, and 
this criterion is the heart of the search engine — it is the distinguishing characteristic that differentiates one 
search engine from another. The ranking criterion must somehow discern which Web pages best respond 
to a user’s query, a daunting task because there might be millions of pertinent pages. Unless a search engine 
wants to play the part of a censor (which most do not), the user is given the opportunity of seeing a list of 
links to a large proportion of the pertinent pages, but with less useful links permuted downward. 
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PageRank is Google’s patented ranking system, and some of the details surrounding PageRank are 
discussed below. 


Facts: 


1. Google assigns at least two scores to each Web page. The first is a popularity score and the second 
is a content score. Google blends these two scores to determine the final ranking of the results that 
are returned in response to a user’s query. 

2. [BP98] The rules used to give each pertinent page a content score are trade secrets, but they generally 
take into account things such as whether the query terms appear in the title or deep in the body 
of a Web page, the number of times query terms appear in a page, the proximity of multiple query 
words to one another, and the appearance of query terms in a page (e.g., headings in bold font 
score higher). The content of neighboring Web pages is also taken into account. 

3. Google is known to employ over a hundred such metrics in this regard, but the details are proprietary. 
While these metrics are important, they are secondary to the the popularity score, which is the 
primary component of PageRank. The content score is used by Google only to temper the popularity 
score. 


63.5 Google’s PageRank 


The popularity score is where the mathematics lies, so it is the focus of the remainder of this exposition. 
We will identify the term “PageRank” with just the mathematical component of Google’s PageRank (the 
popularity score) with the understanding that PageRank may be tweaked by a content score to produce a 
final ranking. 

Both PageRank and Google were conceived by Sergey Brin and Larry Page while they were computer 
science graduate students at Stanford University, and in 1998 they took a leave of absence to focus on 
their growing business. In a public presentation at the Seventh International World Wide Web conference 
(WWW98) in Brisbane, Australia, their paper “The PageRank citation ranking: Bringing order to the Web” 
[BPM99] made small ripples in the information science community that quickly turned into waves. 

The original idea was that a page is important if it is pointed to by other important pages. That is, the 
importance of your page (its PageRank) is determined by summing the PageRanks of all pages that point 
to yours. Brin and Page also reasoned that when an important page points to several places, its weight 
(PageRank) should be distributed proportionately. 

In other words, if YAHOO! points to 99 pages in addition to yours, then you should only get credit for 
1/100 of YAHOO!’s PageRank. This is the intuitive motivation behind Google’s PageRank concept, but 
significant modifications are required to turn this basic idea into something that works in practice. 

For readers who want to know more, the book Google’s PageRank and Beyond: The Science of Search 
Engine Rankings [LM06] (Princeton University Press, 2006) contains over 250 pages devoted to link analysis 
algorithms, along with other ranking schemes such as HITS and SALSA as well as additional background 
material, examples, code, and chapters dealing with more advanced issues in Web search ranking. 


Definitions: 


The hyperlink matrix is the matrix H,,,., that represents the link structure of the Web, and its entries are 
given by 


Pies { 1/|O;| if there is a link from page i to page j, 
ita F 0 otherwise, 


where | O;| is the number of outlinks from page 7. 
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Suppose that there are n Web pages, and let r;(0) denote the initial rank of the ith page. If the ranks are 
successively modified by setting 


i(k 
ri(k 4 pee, RET 2 ina) 
J 


jel; 


where 1; (k) is the rank of page i at iteration k and J; is the set of pages pointing (linking) to page 7, then 
the rankings after the kth iteration are 


r™(k) = (r1(k), r2(k)s - + tn(k)) = 17 (0) H*. 
The conceptual PageRank of the ith Web page is defined to be 
r= jim ri(k), 


provided that the limit exists. However, this definition is strictly an intuitive concept because the natural 
structure of the Web generally prohibits these limits from existing. 

A dangling node is a Web page that contain no out links. Dangling nodes produce zero rows in the 
hyperlink matrix H, so even if limp... H k exists, the limiting vector r’ = limpsoor(k) would be 
dependent on the initial vector r7 (0), which is not good. 

The stochastic hyperlink matrix is produced by perturbing the hyperlink matrix to be stochastic. In 


particular, 
S=H+al!/n, (63.1) 


where a is the column in which 


aa { 1 if page 7 is a dangling node, 
; 0 otherwise. 


S is a stochastic matrix that is identical to H except that zero rows in H are replaced by 17 /n (1 is a vector 
of ls and n = O(10°), so entries in 17 /n are pretty small). The effect is to eliminate dangling nodes. Any 
probability vector p? > 0 can be used in place of the uniform vector. 

The Google matrix is defined to be the stochastic matrix 


G=aS+(l1—-a@)E, (63.2) 
where E = lv? in which v’ > 0 can be any probability vector. Google originally set a = .85 and 
v! = 17/n. The choice of a is discussed later in Facts 12, 13, and 14. 

The personalization vector is the vector v’ in E = lv’ in Equation 63.2. Manipulating v’ gives 


Google the flexibility to make adjustments to PageRanks as well as to personalize them (thus, the name 
“personalization vector”) [HKJ03], [LM04a]. 

The PageRank vector is the left Perron vector (i.e., stationary distribution) 77 of the Google matrix G. 
In particular, 77 (I — G) = 0, where w? > 0 and ||77 ||, = 1. The components of this vector constitute 
Google’s popularity score of each Web page. 


Facts: 


1. [Mey00, Chap. 8] The Google matrix G is a primitive stochastic matrix, so the spectral radius 
p(G) = 1 isa simple eigenvalue, and 1 is the only eigenvalue on the unit circle. 

2. The iteration defined by 77 (k + 1) = m7 (k)G converges, independent of the starting vector, to a 
unique stationary probability distribution 77, which is the PageRank vector. 

3. The irreducible aperiodic Markov chain defined by aw? (k+1) = w"(k)G isa constrained random 
walk on the Web graph. The random walker can be characterized as a Web surfer who, at each step, 
randomly chooses a link from his current page to click on except that 
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(a) When a dangling node is encountered, the excursion is continued by jumping to another page 
selected at random (i.e., with probability 1/1). 


(b) At each step, the random Web surfer has a chance (with probability 1 — a) of becoming bored 
with following links from the current page, in which case the random Web surfer continues 
the excursion by jumping to page j with probability v;. 


4, The random walk defined by r7(k + 1) = r7(k)S will generally not converge because Web’s graph 
structure is not strongly connected, which results in a reducible chain with many isolated ergodic 
classes. 

5. The power method has been Google’s computational method of choice for computing the PageRank 
vector. If formed explicitly, G is completely dense, and the size of n would make each power iteration 
extremely costly — billions of flops for each step. But writing the power method as 


wiktl=7' (KG =an'(k)H + (an (kal? /n + — a)v" 


shows that only extremely sparse vector-matrix multiplications are required. Only the nonzero hi’s 
are needed, so G and S are neither formed nor stored. 

6. When implemented as shown above, each power step requires only nnz(H) flops, where nnz(H) 
is the number of nonzeros in H, and, since the average number of nonzeros per column in H is 
significantly less than 10, we have O(nnz(H)) ~ O(n). Furthermore, the inherent parallism is 
easily exploited, and the current iterate is the only vector stored at each step. 

7. Because the Web has many disconnected components, the hyperlink matrix is highly reducible, and 
compensating for the dangling nodes to construct the stochastic matrix S does not significantly 
affect this. 

8. [Mey00, p. 695-696] Since S is also reducible, S$ can be symmetrically permuted to have the form 


Siu Sia s+ Sir Sirti Siyi. o Bip 
0 Sy +++ Sx Sop4t Sb. Bats She 
P 0 05 ae Se Srr-t1 Sre45 baa S58 
0 0 0 Sr4irtl 0 s+ 0 
0 0 0 0 Sr 42,r-42 be 0 
0 O -:: O 0 0 wile 20h 


where the following are true. 

* For each 1 <i <r, S;; is either irreducible or [0]. 

* For each 1 <i <r, there exists some j > i such that Sij 4 0. 
* p(Sj;) < 1foreach1 <i <r. 

© Srtirtis Sr+2r-+29°*+» Smym are each stochastic and irreducible. 
* lis an eigenvalue for S that is repeated exactly m — r times. 


9. The natural structure of the Web forces the algebraic multiplicity of the eigenvalue 1 to be large. 
10. [LM04a][LM06][Mey00, Ex. 7.1.17, p. 502] If the eigenvalues of S,,., are 


A(S) = {1,1,...,1,Mm-rqis++->Mn}s 1 > |Mm-rgil = +++ = len, 
SS 


mT 
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11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


19; 


then the eigenvalues of the Google matrix G = aS + (1—a@)E are 


(G) = {1, Q,.. £5) (AUm—r41)> + ome »(AUn)}. (63.3) 
Sa 


m—r—-1 


[Mey00] The asymptotic rate of convergence of any aperiodic Markov chain is governed by the 
magnitude of its largest subdominant eigenvalue. In particular, if the distinct eigenvalues A; of an 
aperiodic chain are ordered A; = 1 > |A2| > |A3| = --- = |A,|, then the number of incorrect 
digits in each component of zr? (k) is eventually going to be reduced by about —log10|A2| digits per 
iteration. 

Determining (or even estimating) |A2| normally requires substantial effort, but Equation 63.3 says 
that A2 = a@ for the Google matrix G. This is an extremely happy accident because it means that 
Google’s engineers can completely control the rate of convergence, regardless of the value of the 
personalization vector v! in E = lv’. 

At the last public disclosure Google was setting a = .85, so, at this value, the asymptotic rate of 
convergence of the power method is —logj9(.85) * .07, which means that the power method will 
eventually take about 14 iterations for each significant place of accuracy that is required. 

[LM04a] Even though they can control the rate of convergence with a, Google’s engineers are forced 
to perform a delicate balancing act because while decreasing w increases the rate of convergence, 
decreasing « also lessens the effect of the true hyperlink structure of the Web, which is Google’s 
primary mechanism for measuring Webpage importance. Increasing w more accurately reflects the 
Web’s true link structure, but, along with slower convergence, sensitivity issues begin to surface in 
the sense that slightly different values for a near 1 can produce significantly different PageRanks. 
[KHM03a] The power method can be substantially accelerated by a quadratic extrapolation tech- 
nique similar to Aitken’s A? method, and there is reason to believe that Google has adopted this 
procedure. 

[KHM03b], [KHG04] Other improvements to the basic power method include a block algorithm 
and an adaptive algorithm that monitors the convergence of individual elements of the PageRank 
vector. As soon as components of the vector have converged to an acceptable tolerance, they are 
no longer computed. Convergence is faster because the work per iteration decreases as the process 
evolves. 

[LM02], [LM04a], [LM04b], [LGZ03] Other methods partition H into two groups according to 
dangling nodes and nondangling nodes. The problem is then aggregated by lumping all of the 
dangling nodes into one super state to produce a problem that is significantly reduced in size — 
this is due to the fact that the dangling nodes account for about one fourth of the Web’s nodes. The 
most exciting feature of all these algorithms is that they do not compete with one another. In fact, 
it is possible to combine some of these algorithms to achieve even greater performance. 

The accuracy of PageRank computations is an important implementation issue, but we do not 
know the accuracy with which Google works. It seems that it must be at least high enough to 
differentiate between the often large list of ranked pages that Google usually returns, and since 
a" is a probability vector containing O(10°) components, it is reasonable to expect that accuracy 
giving at least ten significant places is needed to distinguish among the elements. 

A weakness of PageRank is “topic drift.” The PageRank vector might be mathematically accurate, 
but this is of little consequence if the results point the user to sites that are off-topic. PageRank is a 
query-independent measure that is essentially determined by a popularity contest with everyone on 
Web having a vote, and this tends to skew the results in the direction of importance (measured by 
popularity) over relevance to the query. This means that PageRank may have trouble distinguishing 
between pages that are authoritative in general and pages that are authoritative more specifically to 
the query topic. It is believed that Google engineers devote much effort to mitigate this problem, 
and this is where the metrics that determine the content score might have an effect. 
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20. In spite of topic drift, Google’s decision to measure importance by means of popularity over 
relevance turned out to be the key to Google’s success and the source of its strength. The query- 
dependent measures employed by Google’s predecessors were major stumbling blocks in main- 
taining query processing speed as the Web grew. PageRank is query-independent, so at query time 
only a quick lookup into an inverted file storage is required to determine pertinent pages, which 
are then returned in order by the precomputed PageRanks. The small compromise of topic drift in 
favor of processing speed won the day. 

21. A huge advantage of PageRank is its virtual imperviousness to spamming (artificially gaming the 
system). High PageRank is achieved by having many inlinks from highly ranked pages, and while 
it may not be so hard for a page owner to have many of his cronies link to his page, it is difficult to 
generate a lot of inlinks from important sites that have a high rank. 

22. [TM03] Another strength of PageRank concerns the flexibility of the “personalization” (or “inter- 
vention”) vector v’ that Google is free to choose when defining the perturbation term E = lv’. 
The choice of v? affects neither mathematical nor computational aspects, but it does alter the 
rankings in a predictable manner. This can be a terrific advantage if Google wants to intervene to 
push a site’s PageRank down or up, perhaps to punish a suspected “link farmer” or to reward a 
favored client. Google has claimed that it does not make a practice of rewarding favored clients, but 
it is known that Google is extremely vigilant and sensitive concerning people who try to manipulate 
PageRank, and such sites are punished. However, the outside world is not privy to the extent to 
which either the stick or carrot is employed. 

23. [FLM*04], [KH03], [KHG04], [KHMO03a], [LM04a], [LM06], [LGZ03], [LM03], [NZJO1] In spite 
of the simplicity of the basic concepts, subtle issues such as personalization, practical implementa- 
tions, sensitivity, and updating lurk just below the surface. 

24. We have summarized only the mathematical component of Google’s ranking system, but, as men- 
tioned earlier, there are a hundred or more nonmathematical content metrics that are also consid- 
ered when Google responds to a query. The results seen by a user are in fact PageRank tempered 
by these other metrics. While Google is secretive about most of its other metrics, it have made it 
clear that the other metrics are subservient to their mathematical PageRank scores. 
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A signal is a function of time, depending on either a continuous, real time variable or a discrete integer 
time variable. Signal processing is a collection of diverse mathematical, statistical, and computational 
techniques for transforming, extracting information from, and modeling of signals. Common signal 
processing techniques include filtering a signal to remove undesired frequency components, extracting a 
model that describes key features ofa signal, estimation of the frequency components ofa signal, prediction 
of the future behavior of a signal from past data, and finding the direction from which a signal arrives at 
an array of sensors. 

Many modern methods for signal processing have deep connections to linear algebra. Rather than 
attempting to give a complete overview of such a diverse field, we survey a selection of contemporary 
methods for the processing of discrete time signals with deep connections to matrix theory and numerical 
linear algebra. These topics include linear prediction, Wiener filtering, spectral estimation, and direction of 
arrival estimation. Notable ommissions include classical methods of filter design, the fast Fourier transform 
(FFT), and filter structures. (See Chapter 58.) 


64.1 Basic Concepts 


We begin with basic definitions which are standard and can be found in [PM96]. 


Definitions: 


A signal x is a real sequence with element k of the sequence for k = 0,+1,+2,... given by x(k). The set 
of all signals is a vector space with the sum w = x + y defined by w(k) = x(k) + y(k) and the scalar 
product v = ax where a isa real scalar given by v(k) = ax(k). 

A linear time-invariant system is a mapping L(x) = y of an input signal x to an output signal y that 
satisfies the following properties: 


1. Linearity: If L(xo) = yp and L(x) = y,, then L(axo + byy) = ayy + by, for anya,b ER. 
2. Time-invariance: If L(x) = y and & is a shifted version of x given by X(k) = x(k — ko), then 
L(x) = y where p(k) = y(k — ko). That is, shifted inputs lead to correspondingly shifted outputs. 


64-1 


64-2 Handbook of Linear Algebra 


The impulse response of a linear time-invariant system is the output h that results from applying as 
input the unit impulse 6 where 5(0) = 1 and 6(k) = 0 fork # 1. 
The convolution y of h and x is written y = h « x and is defined by the sum 


foe) 


y(k) = $2 A(j)xk — j). 


j=—00 


A signal x is causal if x(k) = 0 fork <0. 

A linear time-invariant system is causal if every causal input gives a causal output. An equivalent 
definition is that a system is causal if its impulse response sequence is causal. 

The z-transform of a signal or impulse response x is 


io.@) 


X(z)= > x(k)z~* 


k=—00 


where X(z) is defined in the region of convergence of the sum. 
The transfer function of a linear system with impulse response h is the z-transform of the impulse 
response 


oe) 


H(z) = S- h(k)z*. 


k=—00 


The discrete time Fourier transform of x is the function of w obtained by evaluating the z-transform 
on the unit circle 
foe) 
Ke) = So xe, 


k=—oo0 


If x is an impulse response of a system, then X(e’) is the frequency response of the system. The discrete 
time Fourier transform is defined only if the region of convergence of the z-transform includes the unit 
circle. 

A filter is minimum phase if the zeros of the transfer function are in the unit circle. 

A finite impulse response (FIR) filter is a system that maps an input signal x to an output signal y 
through the sum 


n-1 
y(k) = S > b(j)x(k = j). 
j=0 


A rational infinite impulse response (IIR) filter is a causal system that maps inputs x to outputs y via 
a relation of the form 


n—1 m 


y(k) = J 7 bi) x(k — 7) — SV ali yk = ji). 


j=0 j=l 


Causality is an essential part of the definition; without this assumption the above relation does not uniquely 
define a mapping from inputs to outputs. 
A signal x has finite energy if 


foe) 


ye |x(k)|? < 00. 


k=—00 


A system mapping input x to output y is stable if |x(k)| < B, for 0 < B, < oo and for all k implies 
that there is 0 < B, < oo such that the output satisfies |y(k)| < B, for all k. 


Signal Processing 64-3 


Facts: 
le 


6. 


8. 


9. 


10. 


The impulse response h uniquely determines a linear time-invariant system. The mapping from the 
input x to the output y is given by y = hx. Convolution is commutative so that ify = g* (hx), 
then y = h x (g * x). Thus, the order in which two linear time-invariant filters are applied to a 
signal does not matter. 

A causal system can be applied in real time in the sense that output y(k) can be computed from 
the convolution y = h « x as soon as x(k) is available. 


. A system defined by its impulse response h is stable if and only if 


ye |h(k)| < 00. 


k=—00 


An expression for the z-transform ofa signal (or an impulse response) does not uniquely determine 
the signal. It is also necessary to know the region of convergence. If X(z) is the z-transform of x, 
then the inverse transform is 


x(k) = aa f xo dz 


wi 


where the integral is taken over any contour enclosing the origin and in the region where X(z) is 
analytic. A system is uniquely determined by its transfer function and the region of convergence of 
the transfer function. 

An FIR filter has impulse response 


b(k) for0<k<n-1 
h(k) = : 
0 otherwise. 
An FIR filter has transfer function 
H(z) = b(0) + b(1)z7! See a b(n = Lz), 


The impulse response h of a causal IIR filter is the unique solution to the recurrence 


0 k<0 
h(k) = ¢ B(k) — YL a(j)h(k— j) O<ksn-1, 
— i a alk — j) k>n 


This recurrence uniquely determines a causal impulse response h and, hence, uniquely determines 
the mapping from inputs to outputs. 
A rational IIR filter has transfer function 


b(0) + b(1)z7! +--+ b(n — Lz-"-) 


H(z) = 1+a(1)z-! +--+ +a(m)z-™ 


A linear time-invariant system with input x and impulse response h with z-transform X(z) and 
transfer function H(z) has output y with z-transform 


Y(z) = H(z)X(z). 


A discrete version of the Paley—Wiener theorem states that x is causal and has finite energy if and 
only if the corresponding z-transform X(z) is analytic in the exterior of the unit circle and 


sup qr |X(rei)|? dw < ©. 


l<r<oo 1 
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If |X(e’”)| is square integrable, then |X (e’”)| is the Fourier transform of some causal system finite 
energy signal if and only if 


- [In (|x(e')|) | dw < ©. 


For rational transfer functions H(z) corresponds to a causal stable system if and only if the poles 
of H(z) are in the unit circle. Thus, FIR filters are always stable and IIR filters are stable when the 
poles of the transfer function are in the unit circle. 


12. A filter that is minimum phase has a causal stable inverse filter with transfer function G(z) = 


1/H(z). That is, filtering a signal x by a filter with z-transform H(z) and then by a filter with 
z-transform G(z) gives an output with z-transform G(z) H(z) X(z) = X(z). The minimum phase 
property is of particular significance since the only filters that can be applied in real time and without 
excessive growth of errors due to noise are those that are causal and stable. In most circumstances 
only minimum phase filters are invertible in practice. 


Examples: 


ie 


The causal impulse response h with h(k) = a* for k > 0 and h(k) = 0 fork < 0 has z-transform 
H(z) = 1/(1 — az7!) with region of convergence |z| > |a|. The anticausal impulse response h 
with A(k) = —a* for k < 0 and h(k) = 0 for k > 0 has z-transform H(z) = 1/(1 — az™!) with 
region of convergence |z| < |a|. 

The causal impulse response h with 


2*43-* fork >0 
0 k<0 


has z-transform 


1 1 2-227} 
at =i 53 To 
1 — (2z) 1 — (3z) 1 — 227-14 22-2 


H(z) = 


with region of convergence |z| > 1/2. The impulse response can be realized by a rational IIR filter 
of the form 


5 1 
y(k) = 2x(k) = Sx(k = 1) + Sy(k = 1) + Zy(k =2). 


The zeros of z* — (5/6)z + (1/6) are 1/2 and 1/3 so that the system is stable. 


64.2 Random Signals 


Definitions of statistical terms can be found in Chapter 52. 


Definitions: 


A random signal or random process x is a sequence x(k) of real random variables indexed by k = 
0,41,42,.... 

A random signal x is wide-sense stationary (for brevity, we will use stationary to mean the same thing) 
if the mean w, = E [x(k)] and autocorrelation sequence r,,(k) = E [x(j)x(j — k)] do not depend 
on j. We assume that all random signals are wide-sense stationary. Two stationary random signals x and y 


are jointly stationary if the cross correlation sequence r,,(k) = E [y(j)x(j — k)] does not depend on j. 
When referring to two stationary signals, we always assume that the signals are jointly stationary as well. 
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A sequence x(k) of real random n x 1 vectors indexed by k = 0,+1,+2,... is stationary if E [x(k)] 
and E [x(k)x'(k)] do not depend on k. 
The autocorrelation matrix of a sequence of stationary random vectors x(k) is 


R, =E [x(k)x" (k)] : 
A zero mean stationary random signal n is a white noise process if it has autocorrelation sequence 


o2 j=0 


ton) = E [n(k)n(k- jf) = 4" 
nn(j) [n(k)n(k — j)] ‘ j 40 
where o? is the variance of n. 

A white noise driven autoregressive process (AR process) x is of the form 


n 
x(k) = n(k) — S°a(j)x(k — j) 
j=l 
where n is a white noise process and the filtering of n to obtain x is causal. 
Given a signal x, we denote the z-transform of the autocorrelation sequence of x by S,(z). The spectral 
density of a stationary random signal x with autocorrelation sequence r,,(k) is the function of w given 


by evaluating S,.(z) on the unit circle 


CO 
Sle) = eae. 
j=-~ 
A random signal with spectral density S,(e!”) is bandlimited if S,.(e'”) = 0 ona subset of [—z, 2] of 
nonzero measure. 
A random signal with spectral density S,(e!”) has finite power if 


/ S,(e!°) dw < 00. 


Tt 


The spectral factorization, when it exists, of a spectral density S,(z) is a factorization of the form 


Sx(z) = L(z)L(z7") 


where L(z) and 1/L(z) are both analytic for |z| > 1. Interpreted as transfer functions L(z) and 1/L(z) 
both describe causal, stable systems. That is, L(z) is causal and stable and causally and stably invertible. 


Facts: 


1. If x(k) is a sequence of vectors obtained by sampling a random sequence of vectors, then under 
suitable ergodicity assumptions 


In computational practice Ry is often estimated using a truncated sum in which n is finite. 

2. The expectation (x,y) = E [xy] can be viewed as an inner product on a real Hilbert space of zero 
mean random variables with E [|x|*] < oo. This gives many results in optimal filtering a geometric 
interpretation in terms of orthogonalization and projection on the span ofa set of random variables 
in a way that is perfectly analogous to least squares problems in any other Hilbert space. From this 
point of view, a white noise process n is a sequence of orthogonal random variables. 
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If a random signal x with autocorrelation sequence r, is passed through a system with impulse 
response h, then the output y has autocorrelation 


ry=her,*h 


where hi is defined by h(j) = h(—j). 

The spectral density is nonnegative, Sx (e?”) > 0. 

The spectral density of white noise is S,,(e’°) = ome 

[Pap85] Ifarandom signal x with spectral density S,.(z) passes through a filter with transfer function 
H(z), then the output y is a stationary signal with 


SA=SQHOHe) ot -S,).= S.A )P. 


If L(z) is the spectral factor of S,(z), then on the unit circle we have S,.(e'”) = |L(e’”)|*. 
[SK01], [GS84] The spectral factorization exists if and only if the signal has finite power and satisfies 


a In (Sx (e’”)) dw > —oo. 


If S,(e*”) is zero on a set of nonzero measure, then the condition is not satisfied. Thus, bandlimited 
signals do not have a spectral factorization. 


If a spectral factor L(z) exists for a signal x with spectral density S,(e!”) and 
CO 1 CO 
_ ~k _ _ -k 
L()= ms I(k)z and M(z) = 7 es s m(k)z-*, 


then the filtered signal n = m x x given by 


foe) 


n(k) = 5° m(j)x(k = j) 


j=0 


is a white noise process with S,,(e'”) = 1. The original signal can be obtained by causally and stably 
filtering n with I: 


lo.e) 


x(k) = So 1(j)n(k = j). 


j=0 


The signal n is the innovations process for x and the representation of x through filtering n by I 
is the innovations representation for x. It is a model of x as a filter driven by white noise. The 
innovations representation is an orthogonalization of the sequence of random variables x. The 
innovations representation for x is analogous to the QR factorization of a matrix. 


Examples: 


1. 


For a white noise process n with variance o,7 = 1, define the autoregressive process x by 
x(k) = n(k) + ax(k — 1) 


where |a| < 1. The impulse response and transfer function of the IIR filter generating x are 


‘° k <0 1 
h(k) = and H(z) = 


ak k>0° l—az’ 
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The autocorrelation sequence for x is the sequence r,, given by 


1 
Txx(k) = ql, 
nt) = 
The autocorrelation sequence has z-transform 
1 
Sx(z) = 
eG ae ae) 
so that the spectral density is 
; 1 
S,(e'?) = : 
hes) 1+ a* — 2a cos() 
The spectral factor is 
1 
L = 
@) 1—az7! 


64.3. Linear Prediction 


Definitions: 


The linear prediction problem is to find an estimate x of a stationary random signal x as a linear 
combination 
foe) 
a(k) = So alj)x(k- j), 


j=l 


with the a(j) chosen to minimize the mean square prediction error E,. = E [le(k)|?] where e(k) = 
&(k) — x(k) is the prediction error. Thus, the goal is to find a current estimate £(k) of x(k) from the past 
values x(k — 1), x(k — 2),.... 

A random signal is predictable if E,, = 0. A signal that is not predictable is regular. 

The order n linear prediction problem is to find an estimate of a stationary random signal x with a 
signal & of the form X(k) = ae an(j)x(k— j) so as to minimize the mean square error E,, = E [|e(k) 7] 
where e(k) = X(k) — x(k). 

A random signal is order n predictable if E,, = 0. 

The prediction error filter and the order n prediction error filter are the filters with transfer functions 


ioe) 
= tae 
j=l 


and 


E,(z)=1—- an(j)z 7. 


Ms: 


j=l 
The Wiener—Hopf equations for the optimal prediction coefficients a(j) are 
tax(1) = s a(j)rxx(l — j) 
j=l 


for! > 1. 
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Given a random signal x(k) with autocorrelation r,,(k), the optimal prediction coefficients a (n)( j) and 
mean square prediction error E,, that satisfy the Yule-Walker equations are 


Txx(O) Txx(1) +e exe (1) ; ; 

Txx(1) Tex(0) : —a,(1) 7 0 
‘ Txx(1) 

Pec) «+ tye(L) yx (0) | LT 4n(7) 0 


or T,a, = Eye}. 
The Levinson—Durbin algorithm for the computation solution of the order n linear prediction 


problem is 
an—-1 4 0 
a, = n 
0 e R an-1 


where R is the permutation that reverses the order of the elements of a,_1. The quantity y, is computed 
from the relation 


tril) — 77) naj nl = 7) 
En-1 


Yn = 


and the mean square prediction error is computed from E, = E,—1(1 — |y,|*). The algorithm starts 
with Eo = rx.(0) and ay = 1. If x(k) is order n predictable, then the process terminates with the order n 
predictor and E,, = 0. 

The parameters y, are known as reflection coefficients, partial correlation coefficients or Schur 
parameters. The complementary parameters are 5, = /1 — |y|?. 


Facts: 


1. [Pap85] The Wold decomposition theorem: Every stationary random signal can be represented as 
the sum of a regular and a predictable signal 


x(k) = x,(k) + xp(k). 


The two components are orthogonal with E [x, (k)xp(k)] = 0. 

2. [Pap85] The Wiener—Hopf equations can be interpreted as stating that the prediction error X(k) — 
x(k) is orthogonal to x(k — 1) for] > 1 in the inner product (x, y) = E[xy]. 

3. The Toeplitz matrix T,, in the Yule-Walker equations is positive semidefinite. 

4, The Levinson—Durbin algorithm is an O(n”) order recursive method for solving the linear predic- 
tion problem. In most cases the numerical results from applying the algorithm are comparable to 
the numerically stable Cholesky factorization procedure [Cyb80]. An alternative fast method, the 
Schur algorithm, computes a Cholesky factorization and can be proven to be numerically stable 
[BBH95]. 

5. The following are equivalent: 


(a) T,—1 is positive definite and T,, is positive semidefinite. 
(b) |yj| < lforl <j <n—Jand|y,|=1. 
(c) Ej > Oforl <j <n—landE, =0. 


(d) The polynomial z”E,,(z) has all zeros on the unit circle and each polynomial zk E,(z) for 
k = 1,2,...,n — has all zeros strictly inside the unit circle. 


6. [Pap85] The prediction errors are monotonic: Ex > Ex4; and Ex > Ew > 0ask > ow. 
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7. [SKO1] Let 
ax(z) = 1 — ax (1)z — a4 (2)z" —- +» — ay (k)z* 

where ax(j) are the prediction coefficients for a signal x with spectral density S,(z). If S,(z) = 

L(z)L(z~") is the spectral factorization of S,(z), then 

1/2 


ee 
BO = (0) 


1 


ay(zt) 


A survey, including other representations of the spectral factor and algorithms for computing it, is 
given in [SKO1]. 
8. [GS84] A finite power random signal x is predictable (i.e., E5. = 0) if and only if 


[ In(S,(e@)) dw > —oo. 


TT 


If the signal is regular, then we have the Kolmogorov—Szegé error formula 


Exo = exp (/ Inte) de) > 0. 


Examples: 


1. Consider the autoregressive process x given by 
x(k) = n(k) + ax(k — 1) 


where n is a white noise process with 07 = 1. The solution to the order 1 linear prediction problem 
is given by the Yule—Walker system 


+ 7 1 Z: E; 
sa | |-a@) of of} 


The Levinson—Durbin algorithm starts with Ey = ry,(0) = 1/(1 — a”) and ap = 1. Thus, 


taeg(1) a/(1— a’) 


Ey 1/(—a?) | 


sn) *-[2]+*[e]-[} 


Thus, a;(1) = a and the optimal linear prediction of x(k) from x(k — 1) is given by 


so that 


K(k) = ax(k — 1). 
The prediction error is given by 


E, = E[&(k) — x(k)] = E[n(k)] = 1. 
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64.4 Wiener Filtering 


Linear optimal filtering of stationary signals was introduced by Wiener [Wie49] and Kolmogorov [Kol41]. 
The material here is covered in books on adaptive filtering [Hay91] and linear estimation [KSH00]. 


Definitions: 


The general discrete Wiener filtering problem is the following: Given stationary signals x and y find a 
filter with impulse response w such that 


a 2 


Ey = E ||y(k)— S> w(f)x(k — 9) 


j=—00 


is minimal. The filter with impulse response w minimizing E,, is the Wiener filter. The goal is to approx- 
imate a desired signal y by filtering the signal x. Equivalently, we seek the best approximation to y(k) in 
the span of the x(j) for —co < j < oo. Note that we leave open the possibility that the Wiener filter is 
not causal. 

Depending on specific relations that may be imposed on y and x, the Wiener filtering problem specializes 
in several important ways: The Wiener prediction problem for which y(k) = x(k+1) where! > 1 and wis 
assumed causal; the Wiener smoothing problem for which y is arbitrary and x is a noise corrupted version 
of y given by x = y + n where nis the noise; and the Wiener deconvolution problem for which we have 
arbitrary desired signals y and x obtained from y by convolution and the addition of noise, x = hx y +n. 
For the Wiener prediction problem, the goal is to predict future values of x from past values. For the 
Wiener smoothing problem, the goal is to recover an approximation to a signal y from a noise corrupted 
version of the signal x. For the Wiener deconvolution, the goal is to invert a filter in the presence of noise 
to obtain the signal y from the filtered and noise corrupted version x. 

The FIR Wiener filtering problem is to choose an FIR filter with impulse response w(k) such that 


n—-1 2 
yk) — 52 w(j)x(k — 9) 
j=0 


is minimal. 
The causal part of a signal x(k) is 


x(k) ifk >0 es 7 
[x(k)]4 = f eeeg %  X@le= yo e% 
Facts: 


1. The stationarity assumptions ensure that the coefficients w(j) that are used to estimate y(k) from 
the signal x do not depend on k. That is, the coefficients w(j) that minimize E,, also minimize 


n—-1 gs 
E || y(k = ko) — S> w(j)x(k = ko — j) 
j=0 


so that the Wiener filter is a linear time invariant system. 
2. Fee Sztz)i= PLE), 


lo,e) 


ry(k) = E [y({)x(G—k)], and S,,(e”)= S- Foyle? 


k=—00 
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If we do not require causality, then the Wiener filter has z-transform 


Syl) 
Wy 


If we do impose causality, then the Wiener filter is 


| Syx(Z) | ms 
T(z) |, L@) 


3. As seen above the Wiener filter depends on the cross correlations r,,(k), but not on knowledge 
of the elements of the unkown sequence y(k). Thus, it is possible to estimate the signal y by 
filtering x without direct knowledge of y. If the autocorrelation sequences rx, (k) and ryy»(k) are 
available (or can be estimated), then r,,(k) can often be computed in a straightforward way. For 
the Wiener prediction problem, r,y.(k) = rxx(/ + k). If the noise n and signal y are uncorrelated 
so that r,,(k) = 0 for all k, then for the Wiener smoothing problem ry, = rx. — fyn. For the 
Wiener deconvolution problem, if g is the inverse filter of h and if y and n are uncorrelated, then 
Lyx = 8 * (1 xx J. Pan). 

4. The coefficients for an FIR Wiener filter satisfy the Wiener—-Hopf equation 


xx (0) Txx(—1) ee Txx(—m aS 1) w(0) yx (0) 
trl) tex) w(1) ryx(1) 
Txx(—1) 

Txx(m — 1) 3 ar Txx(1) Txx(0) w(m — 1) Fyx(m — 1) 


As with linear prediction, this system of equations can be solved using fast algorithms for Toeplitz 
systems. 


Examples: 


1. We consider the first order FIR Wiener prediction problem of predicting y(k) = x(k +2) from the 
autoregressive process x with x(k) = n(k) + ax(k — 1). Thus, we seek w(0) such that 


E [Ix(k + 2) — w(0)x(k)I"] 


is minimal. Since 


1 
Txx(0) = 
OS Tap 
and 
2 
a 
r yx (0) al Txx(2) f= 2 
the Wiener—Hopf equation for w(0) is 
1 a’ 
w(0) = : 
1—a? (0) 1—a? 
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64.5 Adaptive Filtering 


Definitions: 


An FIR adaptive filter is a filter with coefficients that are updated to match the possibly changing statistics 
of the input x and the desired output y. The output is 


m-1 


i(k) = So wel f)x(k = j) 


j=0 


where the w;(j) depend on k. The goal is similar to that of the Wiener filter, that is, to approximate the 
sequence y. The difference is that instead of computing a fixed set of coefficients w(j) from knowledge of 
the statistics of stationary signals x and y, the coefficients are allowed to vary with k and are computed 
from actual samples of the sequences x and y. 

A recursive least squares (RLS) adaptive FIR filter is a rule for updating the coefficients w;(j). The 
coefficients are chosen to minimize 


x(1) x(—1) --» x(l—m+1) wx(0) y(L) 
x~7+1) xl) --- xdl—m-+2) we(1) yl +1) 
min || Dox ; ; . . — Dox 
x(k) x(k—1) +++ x(k—m+1) we(m — 1) y(k) ; 


where k >] +m-—1and D,, = diag(o*", petal ...p°) with 0 < p <1. Equivalently, the vector wx 


is chosen to solve miny, || Dp,4 Ti4we — Dp,cyx||. The parameter p is chosen to decrease the influence of 
older data. This is of use when the signals x and y are only approximately stationary, i.e., their statistical 
properties are slowly time varying. 

A signal x is persistently exciting of order m if there exist c, > 0, c2 > c,, and j such that 


1 
qi< Ap (itis) < cI 


for each eigenvalue A, of Th Tie /(k —1+ 1) and forall k > j. 


Facts: 


1. The vector w; can be updated using using the following recursive formulas: 


Wk = Wk-1 + SK (y(k) = wy_1Xk) 


where 
x(k) 
xX, = : > 
x(k —m-+1) 
ee p> Pr-1Xk ; 
1+ p-2x} Py_ixx 
with 


Py =p? Py-1 — Pp? BeXj Phi. 
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With initialization Py = (Tre Dox Tix) + and we = Py Th D> Yk» the recurrences compute the 
solution w; to the recursive least squares problem. 

2. [GV96] The weight vector w; can also be updated as k increases with an O(n”) cost using a standard 
QR updating algorithm. Such algorithms have good numerical properties. 

3. If x and y are stationary, if p = 1, and if x is persistently exciting, then w;, converges to the 
Wiener filter as k —> oo. Under these assumptions the effect of initializing P;, incorrectly becomes 
a negligible as k — oo. Thus, convergence does not depend on the correct initialization of P, = 
Th Tix. For simplicity, it is common to choose an initial P;, that is a multiple of the identity. 

4. Ifx and y are not stationary, then is typically chosen to be less than one in order to allow the filter 
to adapt to changing statistics. If the signals really are stationary, then choice of p < 1 sacrifices 
convergence of the filter. 

5. There are a variety of algorithms that exploit the Toeplitz structure of the least squares problem, 
reducing the cost per update of w; to O(n) operations [CK84]. An algorithm that behaves well in 
finite precision is analyzed in [Reg93]. 


Examples: 


1. Consider applying a recursive least squares adaptive filter to signals x and y with 


x(1) = 1,x(2) = 2,x(3) = 3,x(4) =4 


and 


y(2) = 5, (3) = 6, y(4) = 8. 


The vector of filter coefficients w3 satisfies 


so that 


We also have 


so that 


P3X4 1 |-4 
24 — aa Le => * 
1+ x4 P3X4 6| 7 
Thus, the next coefficient vector wg is 


10/3 
Wa = ws + 83(y(4) — w3x4) = Gy . 


The process can be continued by updating P4, g;, and ws. 
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64.6 Spectral Estimation 


Definitions: 


The deterministic spectral estimation problem is to estimate the spectral density S,.(e’®) from a finite 
number of correlations r,,(k) for 0 < k <n. Note that this definition is not taken to imply that the signal 
x is deterministic, but that the given data are correlations rather than samples of the random signals. 
The stochastic spectral estimation problem is to estimate S(e’”) from samples of the random signal 
x(k) forO <k <n. 
The problem of estimating sinusoids in noise is to find the frequencies w; of 


&(k) = ay cos(@k + 1) + a cos(wok + 42) +--+ + Ap cos(@pk + np) + wk) = x(k) + n(k) 


where is zero mean white noise with variance o7 and x is the exact sinusoidal signal with random 
frequencies w; € [0, 27) and phases 7;. The frequencies and phases are assumed to be uncorrelated with 
the noise n. The available data can be the samples of the signal x or the autocorrelation sequence rxx(j). 


Facts: 


1. [PM96] A traditional method for the deterministic spectral estimation problem is to assume that 
any rxx(j) not given are equal to zero. When additional information is available in the form of a 
signal model, it is possible to do better by extending the autocorrelation sequence in a way that is 
consistent with the model. 

2. [Pap85] The method of maximum entropy: The sequence rx, (k) is extended so as to maximize the 


entropy rate 
E . 
/ In S(e’®) da, 


where it is assumed that the random signal x is a sequence of normal random variables. It can be 
shown that the spectrum that achieves this is the spectrum of an autoregressive signal 


En 


S(e!”) a 
1 = 224 An( je? 


2? 


where the a,,(j) are the coefficients obtained from the Yule-Walker equations. 
3. [Pis73] Pisarenko’s method: Given a signal comprised of sinusoids in noise, if o,, = 0, then the 
signal sequence x is order p predictable and the signal and its autocorrelation sequence r,, satisfy 


x(k) = a(1)x(k — 1) + a(2)x(k — 2) +--+ + a(p)x(k — p), 
Pxx(k) = a(L)r(k — 1) + a(2)rax(k — 2) + +++ + a(p)rxx(k — p), 


where ap = [1 —a,(1) --- —ap(p)] z is a null vector of the (p +1) x (p +1) autocorrelation 
matrix T = [tj] = [rxx(i — j)]. This is the Yule-Walker system with n = p and E, = 0. When 
On # 0, we have T=T+ oI where 7 is the autocorrelation matrix for #(k). The value of o is 
the smallest eigenvalue of T. Given knowledge of o,,, T can be computed from T = T — 021. The 
prediction coefficients can then be computed from the Yule-Walker equation Ta, = 0. The zeros 
of the prediction error filter 


P 
ZPE,(z) =z? ( = Sacre) 


j=l 
all lie on the unit circle at e’”i for j = 1,2,... p, giving the frequencies of the sinusoids. 
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4, [AGR87] The zeros of a polynomial are often sensitive to small errors in the coefficients. Com- 
puting such zeros accurately can be difficult. As an alternative the zeros of z? E,(z) can be found 
as the eigenvalues of a unitary Hessenberg matrix. Let y; and 6; be the Schur parameters and 
complementary parameters for the autocorrelation matrix R. Then |y;| < 1 forl < j < p—1 
and |yp| = 1. The matrix 


H = G1(y1)G2(y2) +++ Gp (Vp) 


with 


Gj(yj)=1j-18 


ee ® Ip-j-1 


iY; 


is unitary and upper Hessenberg. An explicit formula for the elements of H is 


—y diy, —diSay3 +++ 81 ++ Bp-1p 
8) Viv, —V¥162¥3 t+ —V182+++ 5 p-1¥p 
A= 52 —V2¥3 
bp-1 —Y p-1Yp 


The eigenvalues of H are the zeros of z? E,(z). The eigenvalues of a p x p unitary Hessenberg 
matrix can be found with O(p’) operations. An algorithm exploiting connections with Szegé 
polynomials was given in [Gra86] and stabilized in [Gra99]. A different, stable algorithm involving 
matrix pencils was given in [BE91]. 


Examples: 


1. Consider a stationary random signal with known autocorrelations r,.(0) = landrx,(1) =a <1. 
The other autocorrelations are assumed to be unknown. The Yule—Walker system 


ETL [e 


has solution a,(1) = a and E, = 1 — a’. Thus, the maximum entropy estimate of the spectral 
density S,.(e!”) is 


64.7. Direction of Arrival Estimation 


Definitions: 


The direction of arrival estimation problem can be described as follows. We assume that n plane waves 
sj(k), 1 < j < marrive at an array of m sensors from angles 6; resulting in output x;(k), 1 <1 < m from 
the sensors. We assume that the sensors lie in a plane and that it is necessary to estimate only a single angle 
of arrival for each signal. It is assumed that each sensor output is corrupted by noise m(k), 1 <1 < m. 
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The sensors experience different delays and attenuations of the signal s ;(k) depending on their position 
and the direction of arrival. If the signals are narrow-band signals 


sj (k) = Re (uj(k)erkt"i)) 
for slowly varying u ;(k) and v;(k), then a signal delayed by / is approximately 
(R= DS Re (uj e = Dehra Mer!) a Re (uj (ke Me I") 
Thus, both delay and attenuation of the signals can be described by multiplication by a complex scalar 


dependent on 6;. The observed signals are, therefore, 


n 


x(k) = J © a(6))sj(k) + n(k), 


j=l 

where 

x(k) 

x(k) = : 

Xm(k) 

and a(@) is the complex array response vector. In matrix notation the signal model is 
x(k) = As(k) + n(k), 
where 
A= [a(@,) a(@) --- a(,)]- 


The problem is to estimate the angles 6; from a sequence x(k) of sensor output vectors. We assume that 
the components of the noise vector n(k) are uncorrelated, stationary, and zero mean. A more complete 
description of the problem is given in [RK89]. 

The array manifold is the set of all a(@) for 6 in [—z, z]. 

The signal subspace is the range space of the matrix A. 

The noise subspace is the orthogonal complement of the signal subspace. 

We define the spatial correlation matrices 


R, = E [xx*], R, = E[ss*], and R, = E[nn*] 


where we assume R, is positive definite. These definitions apply directly when dealing with stationary 
random signals. In the deterministic case they can be replaced with time averages. 


Facts: 


1. If the signal and noise are uncorrelated, then 
R, = AR,A* + Ry. 


2. The signal model implies that except for the effect of noise the observation vectors x(k) lie in the 
signal subspace. Thus, in the absence of noise, any n linearly independent observation vectors x(k) 
will give a basis for the signal subspace. The directions 6; are determined by the signal subspace so 
long as the mapping from the set of angles 6; to the signal subspace is invertible. We assume that 
the sensor array is designed to ensure invertibility. 
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3. The signal subspace can be estimated by solving a generalized eigenvalue problem. Then eigenvectors 
associated with the n largest generalized eigenvalues of the pencil 


R, — AR, 


are the basis for the estimated signal subspace. This method can be used to obtain matrices X, and 
X,, such that X, X* is a projection for the signal subspace and X,, X* is a projection for the noise 
subspace. 

4, [Sch79] In order to obtain the angles 6; the MUSIC algorithm searches the array manifold to find 
angles which are close to the signal subspace in the sense that 


M(@) = a*(0)a(9) 

a* (0) X, X*a(@) 
is large. The values of 6 for which M(@) has a peak are taken as the estimates of the angles 6;, 
1 < j <n. The basic approach of the MUSIC algorithm can be used for a variety of parameter 
estimation problems, including the problem of finding sinusoids in noise. 

5. [RK89] The ESPRIT algorithm imposes a structure on the array of sensors. It is assumed that 
the array is divided into two identical subarrays displaced along an R” vector A. Let x;(k) be the 
output of sensor j in the first subarray and y;(k) be the output of the corresponding sensor in 
the second subarray. The displacement A causes a delay in the signal x; (k) relative to y;(k). Ifthe 
signals arriving at the array are again narrow band signals centered at frequency wo, then the signal 
model is 


n 


x(k) = 5° s;(k)a(9;) + ny(k), 


j=l 


n 


y(k) = S- sj (kell sin ))/¢a(g;) + n,(k), 


j=l 


where 6, is the angle of arrival relative to the subarray displacement vector. The combined ESPRIT 
signal model including both subarrays is 


x] _ FA] py [ae 
yk} fae] * Java]? 


where the diagonal matrix © = diag(e’@lAl2sin@/e, |, efollAlla sim(@n)/¢) characterizes the delay 
between the subarrays. 

6. To compute the 6; using the ESPRIT algorithm, we start with the signal subspace associated with 
the ESPRIT signal model estimated as before using a generalized eigenvalue problem. We assume 
that a 2m x n matrix S with columns spanning the signal subspace 


ge 


has been computed and compute the 2n x 2n matrix 


we Vii Vie 
Vo. Vag 
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of right singular vectors of ES Syl Here, V;; and V2 are n x n. The ESPRIT estimates of the 
values e!0llAll2 sin(:)/¢ are the eigenvalues Aj of —Vi2 V55' so that the estimate 6; of 6; is 
6; = sin (c arg(A;)/(I|All2@o)) 


for j = 1,2,...,n. 


Examples: 


1. We consider a single signal arriving at an array of two sensors with array response vector 


en il(at/2-8) 
a(0) = , ; 


If observations of the sensor outputs and solution of the generalized eigenvalue problem Ry —ARn 
suggest that 


X,X*= 


Bes 
ie [use 218 


is the projection for the signal subspace, then the projection for the noise subspace is 


—2i/V3) - 
Bikis | [2i/V3 1/V3]. 


In applying the MUSIC algorithm, we have 


X,X) = 


2 


M(6) = —————-. 
6) 320! 4 1/2 


This function has a maximum of 2/3 at 09 = wz. Thus, the MUSIC estimate of the angle from which 
the signal arrives is 9 = z. 
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Many topics taught in an introductory linear algebra course are often motivated by reference to elementary 
geometry. The geometry is treated as something the student is already familiar with, and reference to it 
is made to give the student a better understanding of algebraic concepts that may be considered, at least 
on first acquaintance, to be rather abstract. What is interesting and important is that the process can be 
reversed: Assuming the linear algebra as known, geometric concepts can be defined and developed on a 
rigorous basis. The use of vector methods in geometry often provides new ways of looking at old problems 
and also helps demonstrate deep and beautiful connections between algebra and geometry. 

This chapter begins by discussing affine geometry, which can be defined, very roughly, as Euclidean 
geometry without any mention of measurement. Thus, affine theorems concern such things as incidence 
and parallelism. An affine space is defined in terms of an action of a vector space V ona set X, pursuant to 
which a vector v acts on a point A of X by sending it to another element v( A) of X, and two points A and 
B of X define a unique vector AB, which acts on A by sending it to B. It might be helpful for the reader to 
think of AB as an arrow starting at point A and ending at point B. This vector then acts on an arbitrary 
point C of X by placing the starting point of the arrow at C and letting the end point of the arrow be v(C). 

By introducing an inner product on a real affine space, Euclidean geometry can be defined. Actually, 
a very interesting mathematical theory can be developed by considering, more generally, an arbitrary 
bilinear form, but in the interest of simplicity, only (positive definite) real inner products are considered 
in this chapter. The general theory is discussed in Chapter III of [Art57] and Chapter 2 of [ST91]. 

Finally, projective geometry is considered. Here the points of the geometry are not vectors but one- 
dimensional subspaces and the resulting incidence properties are different than for Euclidean geometry. 
Projective geometry plays an important role in (among other things) the mathematical theories of elliptic 
curves and algebraic geometry, but these topics are far beyond the scope of this chapter. 

Throughout this chapter, F is a field and V is a finite dimensional vector space over F. 


65.1 Affine Spaces 


Definitions: 


An affine n-space is an ordered pair (X,V) where X is a nonempty set and V is an n-dimensional vector 
space over a field F, which acts on X in the following way: ifve€ V and A € X then there is an element 
v(A) € X, and 
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1. Ifu,v € V and Ae X, then (u+ v)(A) = u(v(A)), and 
2. For any two points A, B € X there exists a unique vector v € V such that v(A) = B. This vector v 
is denoted AB. 


The elements of X are called the points of the affine space (X, V), and sometimes, when no confusion will 
result, one speaks of “the affine space X.” 

A real affine space is an affine space (X, V) where V is a vector space over the field R of real numbers. 

If (X, V) is an affine n-space, W is a (vector) subspace of V of dimension m < n, and A isa fixed point 
in X, then the affine subspace determined by A and W, denoted S(A, W), is the set of all points w(A) as 
w ranges over W. A subset of X is called an affine subspace of dimension m if it is of the form S(A, W) for 
some A in X and subspace W of dimension m (cf. Fact 2, below). 

A one-dimensional affine subspace of X is called a line; a two-dimensional affine subspace, a plane. 

The vector subspace W is called the direction space of the affine subspace S(A, W). 

Two affine subspaces of the same dimension are parallel if they have the same direction space. More 
generally, two affine subspaces (of possibly different dimension) are parallel if the direction space of one 
is a subspace of the direction space of another. 

If C is a point in X, the map P > CB isa bijection between X and V. By identifying a point of X 
with its image in V under this bijection, addition and scalar multiplication in X can be defined so that it 
becomes a vector space isomorphic to V, called the tangent space at C and denoted X(C). 

The elements of a set {X,..., Xa} of d points in an affine space are in general position, or affine- 
independent, if they are not contained in any affine subspace of dimension less than or equal to d — 2. 

Three points that are affine-independent are called noncollinear. 

If A and B are distinct points in a real affine space, the set of points P such that AP = tAB for somet, 
0 < t <1, is called the line segment from A to B and is denoted [.A, B]. The point P which corresponds 
to t =1/2 is the midpoint of the line segment. 

If A, B, and C are three points in general position in a real affine space, then the set of points P such 
that AP = tAB + uAC , where u and t are nonnegative real numbers whose sum does not exceed 1, is 
called the triangle with vertices A, B, and C. 

A subset Y ofa real affine space is convex if whenever A and B are in Y, every point on the line segment 
[A, B] is also in Y. 

If (X, V) is an affine n-space and n > 2, then a bijection T:X — X is semiaffine if the image under T of 
any d-dimensional affine subspace is also a d-dimensional affine subspace, and is affine if it is semiaffine 
and satisfies the following additional condition: If A and B are distinct points and C and D are distinct 
points, and AB = kCD for some nonzero scalar k, then T(A)T(B) = kT(C)T(D). 

If (X, V) is an affine n-space and v € V, the translation T, is the map T,: X — X defined by 
T,(A) = v(A). 


Facts: For proofs, see [ST71]. 

1. If A and B are points in an affine space, then 
(a) AB = 0 ifand only if A = B. 
(b) AB=—BA. 
(c) AB+ BC = AC. 

2. If S = S(A, W) is an affine subspace of the affine space (X, V), then W= {AB: A,B € S}.In 

particular, W is uniquely determined by S. 
3. The affine subspaces S(A, W) and S(B, W) are equal if and only if AB ec W. 


4. The intersection of two affine subspaces is either empty or is an affine subspace. Specifically, if the 
point P is in both S(A, U) and S(B, W), then S(A, U) N S(B, W) = S(P, UN W). 
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10. 


11. 
12. 


13. 


. (Generalized Euclidean Parallel Postulate) If S = S(A, W) is a d-dimensional affine subspace of an 


affine space X, and P is any point of X, then there exists a unique d-dimensional affine subspace 
of X, namely S(P, W), that is parallel to S and contains the point P. 


. Given any two distinct points A and B in an affine space, there is a unique line containing them 


both. 


. The intersection of two convex subsets of a real affine space is convex (possibly empty). In addition, 


every affine subspace of a real affine space is convex. 


. The set of all semiaffine transformations of an affine n-space (X, V) forms a group under the 


operation of function composition. The set of all affine transformations of the affine space is a 
subgroup of this group. 


. Asemiaffine transformation of an affine space of dimension at least two maps parallel subspaces to 


parallel subspaces, i.e, if S and S’ are parallel subspaces of the affine space X and T is a semiaffine 
transformation, then T(S) and T(S’) are parallel. 

If (X, V) is an affine space of dimension at least two and V is a vector space over a field that, like 
the field of real numbers, has no nontrivial automorphisms, then any semiaffine transformation is 
an affine transformation. 

A translation of an affine n-space, where nis at least two, is an affine transformation of that space. 
If {Ao,..., A,} and {Bo,..., B,} are two sets of points in general position in an affine n-space X, 
then there is a unique affine transformation T such that T(A;) = B; for eachi = 0,1,...,n. 

If C is an arbitrary but fixed point of X and T is an affine transformation of X, then T can be 
realized as the composition of a mapping that is a nonsingular linear transformation on the tangent 
space X(C), followed by a translation of X. 


Examples: 


1. 


2: 


If V is any vector space over a field F, then by taking X = V and v(A) = v+ A we obtain 
an affine space, denoted A(V). For two points (i.e., vectors) A and B, AB = B-— A. The affine 
subspaces of A(V) are precisely the additive cosets of the vector subspaces. When V = F”, the 
vector space of column n-tuples of elements of a field F, the resulting affine space is often denoted 
A"(F). 

Any affine subspace S = v+ W of the affine space A( V) is an affine space (S, W), with u(v+ W) = 
(u-+v) + W. It can also be an affine space with a different (but isomorphic) vector space, however. 


For example, in the affine space A?(F), let S be the set of all points P| which satisfy x + y = 1, so 


(S,W) is an affine space with W = { 2) :ueF i But S is also an affine space with associated 


x+ 
ee 
In the affine space A(V), the tangent space V(0) is identical, not just as a set, but also with regard 
to the vector space operations, to the vector space V. Fact 13 therefore implies that any affine 
transformation of this affine space is of the form T(v) = L(v) +b, where L is a nonsingular linear 
transformation from V to itself and b is a fixed vector. 

Thevectors {vo,...,Vn}in-A(V) are affine-independentifand only ifthe vectors{vi—vo,...,Vn—vVo} 
are linearly independent in the vector space V. Thus, for example, if {u, v, w} and {x, y, z} are two 
sets of vectors in general position in the affine space A(V), where V is two-dimensional, then to 
find the unique affine transformation mapping u to x, v to y, and w to z, we first use translation by 
—u to map {u, v, w} to {0, v—u, w— u}, then use the (unique) linear transformation that maps 
the linearly independent vectors v — u and w — u to y — x and z — x, respectively, and finally use 
translation by x. 


vector space F, with the action of F on W given by u P| = | ‘ foru € F. 
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5. Asaspecific illustration of the idea discussed in the previous example, consider A(R). Letu = 
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1 bd 
2 1 1 4 6 . —l 
Vv Hl ,wW 4 Rp < Hl sy Hl ,andz = | . If T denotes translation by - i , then T 
; 0 1 0 : ‘ 
maps u, v, and w, respectively, to u’ = ol? v= ol? andw = il: The linear transformation L 


4, a2 te) OF. . F 
that maps Vv’ and w’ to y-xand z-xis given by the matrix | 3 | . Thus, the composite transformation 


Varta , cote x 3x + 5y —8 O} |3} 45 
LT, which in coordinates is given by P| > b Rape | , Maps u, v, and w to A ) i ) A : 


Composing this composite map with translation by 4| Sives an affine map which maps u, v, w to 


3x +5y—7 


x, y, z. In coordinates, this affine map is given by | > EB +8y—7 


| , which can be viewed as 


; ; —7 
T’L, where T’ is translation by _9 


If {2u, 2v, 2w} is a set of three vectors in general position in the affine plane A?(R), then we can 
think of these vectors as the vertices of a triangle in the Cartesian plane. (The coefficients appear 
simply to make subsequent calculations less messy.) The midpoint of the line containing 2u and 2v 
is u+ v, and similarly for the other sides of the triangle. Thus, the median of the triangle emanating 
from vertex 2w is the line containing 2w and u + v, which is the coset 2w + Span(u + v — 2w). It 
is easy to verify that (1/3)(2u + 2v + 2w) is on this line. Similar calculations show that this point 
is on the other two medians as well. This gives an algebraic proof of the familiar result from high 
school geometry that the medians of a triangle are concurrent. 

Let F be a field that has a nonidentity automorphism f, such as the complex numbers with 
f(x) 
fy) 
transformation that is not affine. To verify that it is not affine, let a be an element of F that is not 
fixed by f, and consider the vectors u = A ,V= i ,we= A , and z = H . It is then easily 


a 
verified that uv = a(wz) but T(u)T(v) = f(a)T(w)T(z). 


complex conjugation. In the affine plane A?(F), the mapping T: P| > | is a semiaffine 


65.2 Euclidean Spaces 


Definitions: 


Euclidean n-space is an affine n-space (X, V) where V isa real inner product space. When n = 2, Euclidean 
n-space is called a Euclidean plane. 

The distance between two points A and B in the Euclidean n-space (X, V) is ||AB ||, where the norm 
is taken pursuant to the inner product in V. This distance is denoted d(A, B). 

Two lines in a Euclidean plane are orthogonal if the vectors that span their respective direction spaces 
are orthogonal with respect to the inner product of V. 

If A and B are distinct points in a Euclidean plane, the perpendicular bisector of line segment [A, B] is 
the line passing through the midpoint of this segment whose direction space is the orthogonal complement 
of the space spanned by the vectorAB. 

An isometry or rigid motion of a Euclidean n-space (X, V) isa bijection T of X that preserves distances: 
d(A, B) = d(T(A), T(B)), for all A and B in X. 

The linear transformation associated with T is the mapping T’:V — V defined by T'(AB) = 
T(A)T(B) (cf. Fact 4, below). 
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The isometry T is direct if T’ has positive determinant, indirect if T’ has negative determinant. 

The remaining definitions in this section apply to the Euclidean plane A?(R) with the ordinary dot 
product as the inner product. (Recall from the preceding section that this is the affine plane obtained by 
taking X = V = R’.) 

A2 x 2 rotation matrix is a matrix of the form Rg = Be ace: | F 

sin 0 cos 6 
: 27% : be sin@ 
A2 x 2 reflection matrix is a matrix of the form Sg = |. ‘ 
sin@ —cosé 

If C is a fixed point in R’, a rotation about point C is a mapping from R’ to itself of the form 
T(X) = Ro(X — C)+ C, for some 2 x 2 rotation matrix Rp. 

If I is a line, the reflection through | is the mapping from A?(R) to itself which sends every point 
on I to itself, and which sends a point C not on / to the unique point C’ with the property that / is the 
perpendicular bisector of the line segment [C, C’]. 

A glide reflection is a reflection through a line / followed by translation by a nonzero vector that spans 
the direction space of 1. 


Facts: For proofs, see [Roe93] and [Ree83]. 


1. The distance function d on a Euclidean n-space is a metric on the set X. Specifically, this means 
that if A, B, and C are arbitrary points of X, then 


(a) d(A, B) is nonnegative, and equal to 0 if and only if A = B. 
(b) d(A, B) = d(B, A). 
(c) d(A, B) < d(A,C) + d(C, B). 


2. The point C is in the line segment [A, B] if and only if d(A, C) + d(C, B) = d(A, B). 

3. (Pythagorean Theorem) If A, B, and C are three noncollinear points in a Euclidean n-space and 
This orthogonal to BC then d(A, B)? + d(B,C)’ = d(A,C)’. 

4, An isometry is an affine mapping. 

5. The associated linear transformation of an isometry is a well-defined orthogonal linear 
transformation. 

6. In the Euclidean space A"(R), when an isometry is written as T(v) = L(v) + b for a nonsingular 
linear transformation L and vector b, the associated linear transformation T’ is simply L. (Such 
an expression for an isometry T is always possible.) 

7. Any translation in a Euclidean n-space is an isometry of that space. The associated linear transfor- 
mation of a translation is the identity map. 

8. The set of all isometries of a Euclidean space (X, V) is a group (under the operation of function 
composition). 

9. The map that associates to every isometry T its associated linear transformation T’ is a homomor- 
phism from the isometry group onto the group of all orthogonal linear transformations of V, the 
kernel of which is the set of all translations. 

10. If! is a line in the affine space A?(R), then reflection through / is an isometry. 

11. If] isa line that passes through the origin in A(R), then reflection through] is a linear transforma- 
tion given by a reflection matrix. Conversely, the linear transformation defined by multiplication of 
a vector by a (fixed) reflection matrix Sp is a reflection through the line passing through the origin 

‘ ane cos(@/2) 

with direction space spanned by ke (6 | . 

12. A rotation is an isometry. 

13. In A?(R), a rotation about the origin 0 is a linear transformation given by a rotation matrix. 

14. In A?(R), the point C is the unique fixed point of a nonidentity rotation about C. Any isometry 
with a unique fixed point is a rotation. 

15. Any isometry of A?(IR) can be written as the product of three or fewer reflections. 
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Every isometry of A?(R) is either a translation, rotation, reflection, or glide reflection. 


. In A?(R), the product of a reflection with itself is the identity mapping. The product of two distinct 


reflections is a translation if the reflections are through parallel lines; if the reflections are through 
two lines intersecting at a point, the product of the two reflections is a rotation about that point. 
The product of three reflections is either a reflection or glide reflection. 

The translations and rotations are the direct isometries of A*(IR) and the reflections and glide 
reflections are the indirect ones. 

Two triangles ABC and A’B’C’ in A?(R) are congruent in the sense of high school geometry 
(i.e., corresponding sides and angles are equal) if and only if there is an isometry of A?(IR), which 
maps A to A’, B to B’, and C to C’. 


Examples: 


1. In A?(R), the line segment between H and a lies on a line with direction space spanned by 


Hl ; the span of - | is the orthogonal complement of this direction space. The midpoint of this 


line segment is AG Therefore, the perpendicular bisector of this segment is the line / given by 


2 —l 
2 x -y F 0 -l 
In A*(R) the map Fe > has matrix e 0 


the line through which this mapping is a reflection, note that its eigenvalues are 1 and —1. An 


| , which is a reflection matrix. To determine 


eigenvector for the eigenvalue 1 is E i . Thus, the line spanned by _ 1 is the line through which 


; ‘ : 0 —l 
this mapping reflects. Alternatively, | _ 


1 0 
—1/V2 
1/J/2| 


The mapping | > - | has matrix k ‘| , which is a rotation matrix (corresponding to 


-y+l 
x 


= $3,/2, so by Fact 11, the line of reflection is 


[a 


spanned by beeen 


0 = 1/2). The mapping T: : => , which is a direct isometry and not a translation, 


must correspond to a rotation also, through a point other than the origin. To determine the center 
of this rotation, we compute the unique fixed point to be x = 1/2 = y. Thus, T is a rotation around 


. Ba 
the point : 
1/2 


65.3 Projective Spaces 


Definitions: 


If V is an (n + 1)-dimensional vector space over a field F , then the n-dimensional projective space based 
on V, denoted P(V), is the set of all subspaces of V. The one-dimensional subspaces of V are the points 
of P(V); the two-dimensional subspaces of V are the lines of P(V). More generally, the k-dimensional 
projective subspaces are the (k + 1)-dimensional subspaces of V. 

When V has dimension 3, P(V) is called the projective plane based on V. 
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A point Span(v) lies on a projective subspace if it is a subset of that projective subspace. 

Relative to a fixed ordered basis of V, any nonzero element of V can be identified with an (n + 1)-tuple 
of elements of F. Thus, by selecting a spanning vector of any point of P(V), that point can be identified 
with an (n + 1)-tuple of elements of F, where not all of the components are zero and where two such 
(n + 1)-tuples are identified if one is a nonzero scalar multiple of the other. Under this identification, the 
(n + 1)-tuple is denoted [a, : ay: ... : dy4,] and called the homogeneous coordinates of the point. (In 
many geometry books, homogenous coordinates are denoted [a),42,...,4n+41], but here that notation 
risks confusion with a 1 x n matrix.) 

When P(V) isa projective space of dimension n, with ordered basis B for V, then for any r-dimensional 
subspace W of V there is an (n+ 1 — r)-dimensional subspace of the dual space V* consisting of those 
linear functionals of V that vanish on W (cf. Section 3.8). In particular, if P(V) is a projective plane, then 
points (respectively, lines) of P(V) correspond to lines (respectively, points) of P(V*). Since, relative to 
the dual basis of V*, any point can be given homogenous coordinates, any line of P(V) can be given the 
homogenous coordinates of its annihilator; these are homogenous line coordinates. 

The dual of a statement concerning points and lines in a projective plane is the statement obtained by 
interchanging the terms “point” and “line.” 

If {A, B,C} and {A’, B’,C’} are two sets of noncollinear points in a projective plane P(V), then the 
triangles ABC and A’B’'C’ are perspective from the point P if and only if the lines containing A and A’, 
B and B’, and C and C’ all pass through the point P. 

The triangles ABC and A’B’C’ are perspective from the line / if the points of intersection of the 
corresponding sides of the triangles all lie on the line /. 

If T is a nonsingular linear transformation of V, then the mapping T from P(V) to itself, de- 
fined by mapping Span({vj,...,Vin}) to Span{T(v1),...;, T'(Win)}, is called the projective transformation 
(or projectivity) determined by T. 

A collineation of P(V) is a bijective mapping from P(V) onto itself which maps subspaces to subspaces 
of the same dimension and which preserves set inclusion. 


Facts: For proofs see [Kap69] or specific references. 


1. Ina projective plane P(V), points and lines satisfy the following incidence properties: 
(a) Two distinct points A and B lie on a unique line, denoted AB. 
(b) Two distinct lines meet in a unique point. 
(c) There are four distinct points, no three of which are collinear. 


2. In the projective plane P(V), the dual of any theorem involving the incidence of points and lines 
is also a theorem. (In the preceding Fact, for example, statement (b) is the dual of statement (a), 
and vice versa.) 

3. [Art57] Any projective transformation of the projective space P(V) is a collineation of P(V). If F 
is a field that does not have any nontrivial automorphisms (such as, for example, the field of real 
numbers), then any collineation of the projective space P(V) is a projective transformation. 

4. In the projective plane P(V), if a point P has homogenous coordinates [a:b:c] and a line | has 
homogenous coordinates [x:y:z], then P lies on/ if and only ifax + by + cz = 0. Thus, to find the 
homogenous line coordinates of the line containing the points [a:b:c] and [d:e: f], we can use the 


x a 
formula for vector cross product to find a vector | y | , which is orthogonal to the two vectors | b 
Zz c 


d 
and | e | under the usual dot product. For an arbitrary field F, this formula yields the homogenous 


i 


line coordinates [x: y: z] of the line containing the two points. 
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(Desargues’ Theorem and its converse) Two triangles in a projective plane P(V) are perspective 
from a point if and only if they are perspective from a line. 

(Pappus’ Theorem) Let A, B,C and A’, B’,C’ be two triples of distinct collinear points in a pro- 
jective plane. Let P, Q, and R denote, respectively, the points of intersection of the lines AB’ and 
A’B, AC’ and C’ A, and BC’ and B’C. Then P, Q, and R are collinear. 

Let A, B,C, and D be four distinct points in a projective plane P(V) with the property that no 
three are collinear. Let A’, B’,C’, and D’ be four other distinct points with this property. Then 
there is a unique projective transformation which maps A to A’, B to B’, C to C’, and D to D’. 


Examples: 


ib 


If V is a two-dimensional vector space over the field of complex numbers, then the projective line 
P(V) can be realized as the set of all homogeneous coordinates [x: y] where x and y are complex 
numbers, not both zero. By identifying [x: y] with the complex number x/y if y is nonzero, and 
with oo if y = 0, we can identify P(V) with the extended complex plane studied in courses on 
complex analysis. Note also that the complex-linear transformation of V given by a nonsingular 


; these are the 


b 
matrix : | represents the mapping of the extended plane given by z > ae - 7 
CZ 


d 


linear fractional transformations studied in such a course. 


. The smallest projective space of dimension greater than 1 is obtained by letting V be a three- 


dimensional vector space over the two-element field Z. In homogenous coordinates, every element 
of P(V) is represented by a triple [a: b: c] where each entry is either 0 or 1 and not all entries are 0. 
Thus, there are seven points in this projective plane, and by duality there are seven lines as well. 
It is easy to verify that each line of this projective plane consists of three points, and each point is 
contained on three lines. For example, the line containing the points [1:1:1] and [1:0:0], obtained 
by taking the span of these vectors, contains these points, the point with homogenous coordinates 
[0:1:1], and no other points (i.e., no other nonzero vector spans a one-dimensional subspace of the 
span of these two points). 
If Vis a three-dimensional vector space over the field of real numbers, then, thinking of the points 
of P(V) in terms of homogenous coordinates relative to a given fixed ordered basis of V, the points 
[0:1:0] and [0:0:1] lie on a unique line, which is the span of these vectors (i.e., the two-dimensional 
subspace of V consisting of all points with homogenous coordinates with first component zero). 
In the dual space of V, relative to the dual basis of the given ordered basis of V, this subspace 
corresponds to the subspace of all linear functionals of V that annihilate all such vectors. This is 
clearly the span of the first vector in the ordered dual basis. Thus, the homogenous line coordinates 
for the line containing the two given points is [1:0:0]. This can also be obtained as a cross product. 
To illustrate Pappus’ Theorem, let six points be given as follows in the real projective plane: A = 
[1:0:0],B =[0:1:0],C = [1:1:0], A’ =[1:1: 2], B’ = [1:2:2],C’ =[0:0: 1]. It 
was observed above that the line BC’ consists of all points with homogenous coordinates with first 
component zero. It is easy to see that the line containing C and B’ is the set of triples of real numbers 
a+b 
ofthe form | a + b | . The intersection of these two lines is obtained by letting a=—b, and, therefore, 
a 
is the point with homogenous coordinates [0:0:1]. By similar calculations, the intersection of the 
lines AB’ and A’B is seen to be the point with homogenous coordinates [1:2:2] and the intersection 
0 
of the lines C A’ and C’ A is the point with homogenous coordinates [0:0:1]. Since the vectors | 0] , 
1 

1 0 

2}, and |0} are linearly dependent, the one-dimensional subspaces spanned by these vectors 

2 1 
are collinear points, as required by Pappus’ Theorem. 
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5. The three incidence relations described in Fact 1 are usually taken as the defining relations for 
an axiomatic definition of projective plane, in which “point,” “line,” and “incidence” are taken as 
undefined notions. It is not the case that any projective plane defined axiomatically in this way is 
of the form P(V) for some vector space V over a field F. As an example, if we interpret “point” 
to mean the set of all ordered pairs of real numbers and “line” to mean any horizontal Euclidean 
line, vertical Euclidean line, Euclidean line of negative slope, or broken Euclidean line of positive 
slope m above the x-axis and slope 2m below the x-axis, we obtain a geometric system that satisfies 
the standard incidence relations for points and lines, and the Euclidean parallel postulate. We can 
convert this example into a projective plane by the addition of “ideal points”: To every line we adda 
new point, with the same point being added to two lines if and only if they are parallel. We also add 
a new line consisting of the ideal points, and no other points. The augmented geometric system 
satisfies the axioms of a projective plane, but neither Desargues’ Theorem nor Pappus’ Theorem 
holds in this plane. (This example is called the Moulton Plane.) 

6. The addition of ideal points, as in the preceding example, can be carried out in any affine plane 

i; 

b 
[a:b:1] and the ideal points are those with homogenous coordinates with a zero in the third 
component. 

7. Ina projective n-space P(V), let H be a subspace of V of dimension n. Starting from an ordered 
basis for H, obtain an ordered basis for V by adding a single vector as the last element of the 
basis. Then, in homogenous coordinates, every point of P(V ) that is not in H has nonzero last 
component. By associating the point with homogenous coordinates [a] : ... : dy+4i] with the 


ai 
An+1 


n-tuple : 


A’(F). Anaffine point can be identified with the projective point with homogenous coordinates 


An 
An+1 


8. Let P(V) be a projective plane, where V is a vector space over a field F that has a nonidentity 
automorphism f. The mapping [a:b:c] +> [f(a): f(b): f(c)] is a collineation. Since this 
mapping fixes the points [0:0:1], [0:1:0], [1:0:0], and [1:1:1] and is not the identity, it cannot, by 
Fact 7, be a projective transformation. 

9. Inthe real projective plane, consider the six points A= [1:2:1], B = [2:0:1], C = [5:5:1], A’ = [2:4:1], 
B’ = [4:0:1], and C’ = [10:10:1]. Identifying the projective point with homogenous coordinates 

a 

b 

diagram that the triangles ABC and A’ B’C’ are perspective from the origin, since all lines AA’, BB’, 

and CC’ pass through this point. Thus, in the real projective plane these triangles are perspective 
from the point [0:0:1]. Desargues’ Theorem, therefore, asserts that the points of intersection of the 
lines ABand A’ B’, ACand A’C’, BCand B’C’ willlie ona line. A simple calculation shows that these 
three points, in homogenous coordinates, have third component zero, so these points are indeed 
collinear. This corresponds to the fact that as Euclidean lines there are no points of intersection — 
the corresponding sides of the triangles are parallel in pairs. The points with third component zero 
are the “ideal points” that have been added to Euclidean geometry to form the real projective plane. 


| we can think of the set of points that are not in H as the affine n-space A"(F ). 


[a:b:1] with the ordinary Euclidean point , as in Example 6 above, it can be seen by a simple 
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This chapter presents some facts and examples illustrating the interplay between matrix theory, graph 
theory, and n-dimensional Euclidean geometry. The main objects of Euclidean geometry are, of course, 
the points. The simplest way of introducing them is to identify points with the endpoints of vectors; 
formally, one introduces an artificial point, the origin O, and the points are sums of this origin with any 
vector. A more general treatment of Euclidean n-space can be found in Chapter 65. 


66.1 Euclidean Point Space 


Definitions: 
An arithmetic Euclidean vector space is the real inner product space R” with the standard inner product 
(xy y) = y'x. 

The arithmetic point Euclidean n-space E” based on the vector space R” has as points the column 
n+ 1-tuples with last coordinate 1, e.g., C = [c1,€2,...5 ns 1]’, and as vectors the column n + 1-tuples 
with last coordinate 0, e.g., v = [V1,V25..-5Vn» o]?. 

The origin O is the point [0,...,0, 1] T; the numbers c;,...,¢, are coordinates of the point C. 

Algebraic operations are defined with points: If Ay, Aj,..., A, are points, a), a2,...,a, real numbers, 
then the symbol 


a, Ay + a2A2 + +++ + AmAm 


means a point if and only if )> a, = 1, and a vector if and only if 5) a, = 0. In no other case is this 
symbol defined. 
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The Euclidean distance, or distance, of two points A and B is the length of the vector A — B, i-e., for 
A= [a1,a2,. +9 Any ie and B = [by, bo, . on Dn, 1], 


A — Bil = V(r — bi)? +e + (an — bn). 


Let S = {Aj, Ao,..., Am} be a set of points in E”. The point 


C= a A; + a,A. +---+4,Am and Soa =; 
k=1 


is a linear combination of S. The set of all linear combinations of S is the linear hull of S, denoted by 
L(S). 

A point C is linearly dependent on S if C € £(S). The set S is called linearly dependent if there is 
a point in S that is linearly dependent on the remaining points. Otherwise, the set S is called linearly 
independent. 

Linear hulls of systems of points are called linear subspaces of E”. 

A linear subspace M has dimension k if the maximum number of linearly independent points in M is 
k+l. 

Linear subspaces of dimension 1 in E” are called lines. 

Linear subspaces of dimension n — 1 in E” (i.e., of codimension 1) are called hyperplanes. 

If@ isa hyperplane in E” defined by the equation a1x; + 2x2 +++: +QnXn +o = 0 (cf. Fact 10), the 
vector u = [@,...,@,,0]" is a normal vector to @. 

The distance of point C to a hyperplane @ is the minimum of all distances of the point C from the 
points in the hyperplane a. 

Two hyperplanes are called parallel if their normal vectors are proportional, i.e., one is a scalar multiple 
of the other. They are called orthogonal (or perpendicular) if their normal vectors are orthogonal. 

Let S = {Aj, Ao,...,Am} be a set of points in E”. Then the set of all points of the form 


aA, Te az Az Feaoean amAm> 


for which all the coefficients a; are nonnegative and > ay = 1, is called the convex hull of S; we denote 
it by C(S). 

The convex hull of two distinct points A, B is called the segment and is denoted by AB. 

The point A + 5B is the midpoint of the segment AB. 

A set S € E" is convex if S has the property that with any two points A and B in S, all points of the 
segment AB are in S. 

A hyperplane @ with normal vector [@,@,...,@,] determines two open halfspaces; one is the set of 
all points X = [x,,x2,..., Xn) 1)" satisfying 


OX] + OQX2 +--+ + AnXy +A > 0, 


while the other is the set of all points satisfying the reverse inequality. 
In addition, one can speak about closed halfspace if in the inequality, equality is also admitted. 


Facts: 
These facts follow from facts in Chapter 1 and Chapter 65. 


1. Points and vectors in arithmetic Euclidean n-space satisfy the following: 
(El.) The sum ofa point and a vector is a point. 
(E2.) The difference of two points is a vector. 


(E3.) (C+v) +w= C+ (v+ Ww), where C is a point and v, w are vectors. 
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10. 


11. 


12: 


13. 
14. 


. Arithmetic Euclidean n-space, with vectors acting on points by addition, is a Euclidean n-space as 


defined in Chapter 65. 


. A linear hull is an affine subspace as defined in Chapter 65. 
. The set S = {Aj, Ay,...,Am} is linearly independent if and only if A, 1) +) = din 0 


are the only numbers A,,...,A,, for which the zero vector 0 satisfies 0 = A,A; + A2A. +--+ + 
Ap bog and “Soe Ag 0s 


. The set S = {Aj, Ay,..., Ay} is linearly independent if and only if every point P € £L(S) has a 


unique expression as 


P=), A; + ApAd b+ + AA and Sok =1 
k=1 


. A linearly independent set in E” contains at most n + 1 points. A linearly independent set with 


n+ 1 linearly independent points in E” exists. 


. Any linearly independent set has the property that each of its nonempty subsets is linearly inde- 


pendent as well. 


. Let the set S consist of m points A, B, ..., G defined by 


A= [aiinccd Gal vB S [Bish ais Bll 23 G= [gixgive sail’. 


Then S is linearly independent if and only if the m x (n + 1) matrix 


a, a... Gy 1 

b, by... by 1 

Zi G2 ee Sn 1 
has rank m. 

Let A = [4),4,...,41]', B = [b1,b2,...,b,,1]', ..., EF = [fisfsese fel] * be n 
linearly independent points in E”. Then the linear hull of these points consists of all points 
X = [x1,X2,...5X,, 1]? in E” which satisfy the equation 

Kyo “Kei 244." SKK 1 ] 
a, a ... Gy 1 
det |b; by b, 1] =0. 
fi fe oss fy 1 
A hyperplane can be characterized as the set of points K = [x1,%2,..-,Xn. 1]? such that the 


coordinates satisfy one linear equation of the form 
Oy X, Hx. +--+ + AyX, +A = 0, 


in which not all of the numbers a1, @,...,@, are equal to zero. 

We can generalize linear independence as well as the equation of the hyperplane in Fact 9 by 
including cases when some of the points (but not all of them) are replaced by vectors. We simply 
put into the corresponding row the (m + 1)-tuple a), a2,...,a,,0 instead that for the point. 

Two parallel but distinct hyperplanes have no point in common. Conversely, if two hyperplanes in 
E",n > 2, have no point in common, then they are parallel. 

Hyperplanes are parallel if and only if they are parallel as affine subspaces as defined in Chapter 65. 
The definitions of segment and midpoint are equivalent to the definitions of these terms in 
Chapter 65 for arithmetic Euclidean n-space. 
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15. The distance of a point C = [c1,¢2,...,Cn, 1]? from a hyperplane @ given by equation ax; + 
OnX. $e + AyX, +A = 0 is 


Joep Cy + OC + +++ + AnCy + Al 


Vai top t---+a2 


16. The distance ofa point C = [c¢1,¢2,...,Cn, 1]? from ahyperplane @ given by equation a; x; +a2x2.+ 
+++ 4+0,Xn +9 = Ois the distance of C from the point F = C— yu, where u = [a1,02,...,@,,0]" 
is the normal vector to @ and 


QC, + O2C2 +++> + AyCy +A 
Ord Os brett ae 


Examples: 


1. In E3, the points A; = [2,—-1,2,1]7, Ay = [1,1,3,1]7, and As = [0,0, 1,1]? are linearly inde- 
pendent since the rank of the matrix 


2 -1 2 1 
1 13 1 
0 01 1 


is 3. 

2. The point A, = [2,0,3, 1]? is linearly dependent on the points A, A2, and A; from Example 1, 
since Ay = FAy + 5 Ap _ 5 A3. 

3. The equation of the hyperplane (which is a plane) determined by the points in Example 1 is 


xX, XxX, x3 1 
2 -1 2 1 

0 = det i i) a4 = —3x, — 3x2. + 3x3 — 3. 
0 0 1 1 


4. The triangle with vertices A;, A2, A3 from Example 1 is the convex hull of the three points Aj, Ay, 
and A3. 
5. Let n > 3 be an integer. In the Euclidean n-space E", define points 


F, = [n—k,n—k,...,n —k,—-k,—k,...,—-k,1], k=1,...,n; 
SS ee 
k—times (n—k)—times 


Fy = [n—1,n—2,...,0,1]7, and 


Observe first that the points C, F,, F:,..., F,, arelinearly dependent since C = +F, + +Fo+- sof +F,,. 
On the other hand, the points Fo, F,,...,F, form a linearly independent set by Fact 8 since the 
determinant (of an (1 + 1) x (n + 1) matrix) 


n—-1l1 n-2 n-3 n-A4 0 1 
n—1 1 1 1 —1 1 
n—-2 n-—2 2 2 —2 1 

det aoe 
2 2 2 2 —(n—2) 1 
1 1 1 1 —(n-1) 1 


Oo 
Oo 
Oo 
Oo 
Oo 
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is different from zero: Subtracting the +-multiple of the sum of the second till last row, from the first 
row, the first row is a $(n — 1)-multiple of the row of all ones. Factoring out this number s(n —1) 
from the determinant, subtracting the resulting first row from the nth row, the 2-multiple of the 
first row from the (1 — 1)st row, etc., till the (n — 1)-multiple of the first row from the second row, 
we obtain the determinant of an upper triangular matrix with all diagonal entries different from 
zero. The value of the determinant is then also easily determined. 

The Euclidean distances between the points F; and F;,),i = 1,..., — 1, as well as between 


E,, and F;, are all equal, equal to /n(n — 1). The point C has same distances from all points F,, 
i= 1,2,...,n, equal to ,/5(n— In(n+ 1). 

All the points F;,i = 1,2,...,n, as well as the point C;, are points of the hyperplane H with 
equation )>"_, x; = 0. 

The vector Fy — C is the i(n — 1)-multiple of the vector u = [1,1,...,1]7 (which is throughout 
denoted as 1). This vector is at the same time the normal vector to the hyperplane H. It follows 
that the distance of the point Fy from H is equal to the length of the vector Fo — C, which is 


$(n — 1)./n. The same result can be obtained using Fact 15. 


The hyperplane with equation 
Xp tx. +--+ +x, -1=0 
is parallel to H; the hyperplane 
xX) —%2.=0 


is orthogonal to H since the vector [1,—1,0,...,0, 0] T is orthogonal to the vector u. 


66.2 Gram Matrices 


Definitions: 


The Gram matrix G(S) of an ordered system S = (aj,a2,...,@,) of vectors in the Euclidean vector 
n-space R" is the m x m matrix G(S) = G(aj,a2,...,4m) = [(aj, aj) ]. (See also Section 8.1.). 

Let O, Aj, Ao,..., Ax, k = 2, be linearly independent points in E", and let u, = A; — O,w. = 
A, — O,...,uz = Ax — O, be the corresponding vectors. We call the set of all points of the form 
O+ ey axuz, where the numbers a; satisfy 0 < a; < 1,1 = 1,...,k, the parallelepiped spanned 
by the vectors u;. For k = 2, we speak about the parallelogram spanned by u, and up. 

If uy, U,...,U, is a basis of an n-dimensional arithmetic Euclidean vector space and vj, V2,...,V, isa 
set of vectors such that the inner product of u; and v; is the Kronecker delta 5;;, then this pair of ordered 
sets is a biorthogonal pair of bases. 


Facts: 


Facts for which no specific reference is given follow from facts in Chapter 1. 


1. The Gram matrix is always a positive semidefinite matrix. Its rank is equal to the dimension of the 
Euclidean space of smallest dimension which contains all the vectors of the system. 

2. Every positive semidefinite matrix is a Gram matrix of some system of vectors S in some 
Euclidean space. 

3. Every linear relationship among the vectors in S is reflected in the same linear relationship among 
the rows of G(S), and conversely. 

4, The k-dimensional volume of the parallelepiped spanned by the vectors 


U),Uy,..., UX is 


/det G(uj,up,..., uz). 
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To every basis uy, U2,...,U, of an n-dimensional Euclidean vector space there exists a set of vectors 
V),V2;... )V, such that this pair is a biorthogonal pair of bases. The set of the vjs is also a basis and 
is uniquely determined. 

If both bases in the biorthogonal pair coincide, the common basis is orthonormal, and an ortho- 
normal basis forms a biorthogonal pair with itself. 

The Gram matrices G(u,up,...,u,) and G(v),v2,...,Vv,) of a pair of biorthogonal bases are 
inverse to each other: 


G(uy, U,. . .)U,)G(Vj1, V2. . 15 Vn) =I. 


[Fie64] Let A = [a;;] be a positive semidefinite matrix with row sums zero. Then 
2max /dii < ) J aii- 
1 . 
4 


[Fie61b] If A is positive definite, then the matrix Ao A~! — I is positive semidefinite and its row 
sums are equal to zero. 
If A = [aj;] is positive definite and At= [a;;], then 


a;j;; > 1 for all i 
and 


2 max Vaio -l< So aici —1. 
i 


[Fie64] Let A = [a;,] be a positive definite matrix, and let A~! = [ax]. Then the diagonal entries 
of Aand A“ satisfy the first condition in Fact 10 and 


2 max(./ajioii — 1) < SS Wana — 1). 


1 


Conversely, if some n-tuples of positive numbers a;; and a;; satisfy these conditions, then there 
exists a positive definite n x n matrix A with diagonal entries a;; such that the diagonal entries of 
A7! are aj;. 

[Fie64] Let the vectors uj, Up,..., Uy, V1, V2... V_ forma pair of biorthogonal bases in a Euclidean 
n-space E. Then 


luli] = 1, 2=1,...,2, 
2max(lluillivill — 1) =) (lei llllvill =D. 
4 
Conversely, if nonnegative numbers @),@,..., On, 81, Bo,..-5 By Satisfy 
ajpj>1, i=1,...,n, 
2 max(qj 6; -l1)< S "(abi — 1), 


1 


then there exists in E" a pair of biorthogonal bases uj, v;, such that 
lull =o; |Ilvill = 6, t=L...,n. 


[Fie64] Let A = [a;;] be ann x n positive definite matrix, n > 2, and let Atl= [a;;]. Then the 
following are equivalent: 
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(a) f@anQ@in — 1 = 202) (fanaa — 1). 
(b) 2 = —* _, i,j =1,...,n—1,and 


Gi 4 jj oii 04; ” 
l,...,n—-1. 


Gin Qin 
> 
Gii JS Ann Qi SAnn 


(c) A is diagonally similar to 


where ¢ is a real vector with n — 1 coordinates and 


V1+cle—1 


o= T 


cc 


ifc 4 0;ifc=0,0 =0. 


14. To realize a positive definite n x n matrix C as a Gram matrix of some n vectors, say a), a2,...,@n5 
it suffices to find a nonsingular matrix A such that C = AA? and to use the entries in the kth 
row of A as coordinates of the vector a,. Such matrix A can be found, e.g., by the Gram—Schmidt 
process (cf. Section 5.5). 


Examples: 
1. The vectors u, = [1,3]?, w = [2,—1]" in R? are linearly independent and form a basis in R’. 
10-1 
The Gram matrix G(u;, uw.) = in is nonsingular. 


2. To find the pair of vectors v), v2 that form a biorthogonal pair with the vectors uj, u, in Example 1, 
observe that v, should satisfy (u;,v,;) = 1 and (u:,v,) = 0. Set v) = [x1,x2]7; thus x; + 3x2 = 1, 


2x1 — x» = O,ie., vy = [;, 27, Analogously, v. = (2,-4 T The Gram matrix is G(v), v2) = 
5 1 
F : . It is easily verified that G(vj, v2) is the inverse of G(uj, uz), as stated in Fact 7. 
49 «49 
3. The pairs of vectors (e; = [1,0]7, ey = [0,1]") and (u, = (2, 2\’,w = (+, 2]") are ortho- 


normal bases for R’, so G(e),e)) = I and G(uy, u) = J are inverses, but (e1, e2) and (uj, uy) are 
not a biorthogonal pair of bases. 


66.3 General Theory of Euclidean Simplexes 


An n-simplex in E” is a generalization of the triangle in the plane and the tetrahedron in the three- 
dimensional space. Just as not every triplet of positive numbers can serve as lengths of three sides of a 
triangle (they have to satisfy the strict triangle inequality), we can ask about analogous conditions for the 
simplex. 


Definitions: 


An n-simplex is the convex hull of n + 1 linearly independent points in E”. 

The points are called vertices of the simplex. The convex hull of a subset of the set of vertices is called a 
face of the simplex. 

If the subset of vertices has k + 1 elements, the face has dimension k. One-dimensional faces are called 
edges. 

If Aj, Ay,...,A,41 are the vertices of an n-simplex X, we write D = {Aj, Ap,...,An41}. The (n — 1)- 
dimensional face opposite A; is denoted by w;. A vector n is an outer normal to @; ifn is a normal to a; 
and for C € w;, C + nis not in the same halfspace as Aj. 

The dihedral interior angle between w; and w;,i ¢ j (opposite the edge A;A;) is 7—(the angle 
between an outer normal to @; and an outer normal to ;), and is denoted by ¢;;. 
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The barycentric coordinates with respect to the simplex © = {A),A),...,Ay+4i1} of a point P in 
L£(Aj,Aa,...,An41) are the unique numbers A; such that P = A,A; + AzAy +--+ + Angi Angi and 
Mu Ax = 1 (cf. Fact 5 of section 66.1). Strictly speaking, these numbers are the inhomogeneous 


barycentric coordinates. 

The homogeneous barycentric coordinates of 4A; + AzA) + +++ + Anyi An41 are the numbers A; 
(not all 0). The expression 4,A; + AzAy +--+ + Anyi An+1 Means a proper point if eS AR #0 
(namely the point with inhomogeneous barycentric coordinates pi ,..., PAn+1 with p = ( a AR), 
or an improper point (determined by the (nonzero) vector A, A; +A2Ag +--+ -+An41An41) if Ap = 0. 
These coordinates can be viewed as in projective n-dimensional geometry (see Chapter 65). The set 
of improper points is, thus, characterized by the equation Sie Ay = 0 in homogeneous barycentric 
coordinates, and is called the improper hyperplane. 

The circumcenter of a simplex is the point that is equidistant from the all vertices of the simplex. The 
set of all points at that common distance from the circumcenter is the circumscribed hypersphere. 

The (1 + 1) x (n + 1) matrix M = [mjj] with mj; equal to the square of the distance between the ith 
and jth vertex of an n-simplex & is called the Menger matrix of X. 

The Gramian of an n-simplex © = {Aj,Ao,...,An41} is defined as the Gram matrix Q of n+ 1 
vectors Nj,M),...,M,+4; determined as follows: The first n of them form the biorthogonal pair with the 
n vectors uy = Ay — Ayyj,U. = Ay — Anyi,...,U, = A, — Anyi, the remaining n,¥, is defined by 


n 
May = — Dj Mi. 


Facts: 
1. There are ("3") edges {A;, Aj}, i # j, in an n-simplex. 
2. A2-simplexisasegmentand its circumcenter is its midpoint. The circumcenter ofan n-simplexis the 
intersection of the lines ¢;, where @; is the line normal to face @; and through the 


circumcenter of @;. 


3. [Blu53] (Menger, Schoenberg, etc.) The numbers mj;, i, j = 1,...,+ 1, can serve as squares of 
the lengths of edges (i.e., of distances between vertices) of an n-simplex if and only if m;; = 0 for 
alli, and 


y mjjx;x; <0, whenever y x; = 0. 


i,j i 


4. (Consequence of Fact 3) The matrix 


where 1 is the column vector of all ones and M = [mj;] isa Menger matrix, is elliptic, i.e., has one 
eigenvalue positive and the remaining negative. 

5. The Menger matrix is the matrix of the quadratic form )> i,j MijXiXj- Ifx,,...,Xn41 are considered 
as homogeneous barycentric coordinates of the point X = [x),..., X41], then the equation 


; MjjXiXj =0 
i,j 


is the equation of the circumscribed hypersphere of the n-simplex Y. (Thus, the condition in Fact 
3 can be interpreted that for all improper points, the value of the quadratic form ae, Mj xX; X; is 
strictly negative.) 
6. The n-dimensional volume V of an n-simplex © satisfies 
_ ey 


Vv? = ~—— det M, 
2"(n!)2 > 


where Mp is the matrix in Fact 4 using the Menger matrix M of &. 
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10. 


11. 


The volume V, of the n-simplex with vertices in O, Aj,...,A, is equal to V, = Vp, where V, 
is the volume of the parallelepiped defined in Fact 4 of section 66.2 by n vectors A; — O. Thus, 
Y= 4 /det G(u,...,U,), where G(...) means the Gram matrix and uy, = A; — O. 


T 
foo 4 . F : . : 
Let Qo = . , where the matrix Q is the Gramian of the n-simplex , the numbers in 


qo 
the column vector qo are (—2)-multiples of the inhomogeneous barycentric coordinates of the 
circumcenter of ©, and ; Goo is the radius of the circumsphere of &. Then Mp Qo = —21. 
The vectors nj, m,...,N,+1 from the definition of the Gramian are the vectors of outer normals 
of the simplex, in some sense normalized. 
The Gramian of every n-simplex is an (n + 1) x (n + 1) positive semidefinite matrix of rank n 
with row sums equal to zero. Conversely, every such matrix determines uniquely (apart from the 
position in the space) an n-simplex in E” whose Gramian this matrix is. 
Let Us be a face of an n-simplex & determined by the vertices with index set S C {1,..., + 1}. 
Let Q be the Gramian of X=. Then the Gramian of Ys is the Schur complement Q/Q(S), where 
Q(S) denotes the principal submatrix of Q corresponding to indices in the complement of S. 


Examples: 


1. 


Let us find the Menger matrix and the Gramian in the case of the segment {Aj, Az} of length 
0 d 


d (i.e., a 1-simplex). The Menger matrix is F | .Ifu,; = A; — Ay, then ny = mall since 


| . Indeed, 


(u,,n,) has to be 1. Thus, the Gramian is the Gram matrix G(n;, —nj), ie., 


0 1 1) fd -1 -1 

1 0 @ 1 a 1 | = —2]s, as asserted in Fact 8. 

1 @ oO} |-l -z &# 
Consider the simplex © = {A; = [0,0, 1]7, A> = [1,0,1]7, A3 = [1,2,1]7} in E?. We show how 
Fact 6 and Fact 7 can be used to find the volume V of this simplex. 


To use Fact 6, compute the squares of the distances between the vertices to obtain the Menger 
011i 


matrix: M = . Since det My = —16, V = 1 by Fact 6. 


nN 


0 1 
1 0 
5A 


eS Se 
Oo 


1 1 
To use Fact 7, compute the vectors u; = [1,0,0]? and uz = [1,2,0]", so G(w, uw) = f ] Ce 


Fact 7, V = 3/det G(uy, wy) = 3/4 = 1. 


In this example, the volume can be found directly by finding the area of the triangle. 


. Consider the simplex in Example 2. Let us compute the Gramian to illustrate Fact 8 for this simplex. 


Since u; = A; — As = [—1,—2,0]7, w = Ay — Aj = [0,—2,0]’, the vectors n),n, forming 


a biorthogonal pair with u,, uy are (as in Example 2 of Section 66.2) n; = [—1,0, 0]',m = 
1 -l 0 
[1,-4,0]”. Thus, n3 = [0, $,0]” and G(nj,m,n3) = |—1 oA 


BIR RIE 


0 — 
The line ¢; is the set of points of the form [x, 1,1], and line @; is the set of points of the form 
[5 y> 1]", so the circumcenter of the simplex is ¢ = [5 1,1)". The (inhomogeneous) barycentric 
coordinates b,, bz, b3 of c can be found by solving b; A; + by A) + b3A3 = cb) + bo + b3 = 1 
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to obtain b, + by 0, b3 $e The radius of the circumsphere is =) SO goo = 5. Thus, 
5 —-l 0 -l 
a es es a es 
Qo = i ; P Lf which is indeed equal to the matrix —2M) ~. 
“2 4 4 
1 0-5 3 


4. We can use either Example 1 or Fact 11 and Example 3 to find the Gramian of Ys for {2, 3} (again 


ols. ere de =k 
with & defined in Example 2). By Example 1, the Gramian is a . = F : 
& a 4 4 
66.4 Special Simplexes 
Definitions: 
Let us color an edge {A;, Aj} of an n-simplex with vertices Aj,...,An41, 1 = 2, in: red, if the opposite 


interior angle ¢;; is acute; blue, if the opposite interior angle ¢;; is obtuse; it will stay uncolored, if the 
opposite interior angle ¢j; is right. 

The edge {Aj;, Ax} is called colored if it is colored red or blue. The assignment of red and blue colors is 
a coloring of the simplex. 

The graph of the n-simplex is the graph with vertices 1,2,...,-+ 1 and edges {i,k}, i 4 k, for which 
{Aj, Ax} is colored. The colored graph of the simplex is the graph of the simplex colored in red and blue 
in the same way as the corresponding edges of the simplex. 

An n-simplex is called hyperacute if each of its dihedral interior angles ¢;; is either acute or right. 

A hyperacute n-simplex is called totally hyperacute if its circumcenter is either an interior point of the 
simplex, or an interior point of one of its faces. 

Let C denote the circumcenter of the n-simplex with vertices Aj,...,An+41. We extend the coloring 
of the simplex as follows: We assign the segment CA, the red color if the point C is in the same open 
halfspace determined by the face w, as the vertex A;, and the blue color, if C is in the opposite open 
halfspace. We do not assign to CA, any color if C belongs to w,. This is the extended coloring of the 
simplex. 

In the same way, we speak about the extended graph and extended colored graph of the simplex, adding 
to n + 1 vertices another vertex n + 2 which corresponds to the circumcenter. 

A right simplex (cf. [Fie57]) is an n-simplex which has exactly n acute interior angles and all the 
remaining (5) interior angles right. 

In a right simplex, the edges opposite the acute angles are called legs. The subgraph of legs and their 
endpoints is called tree of legs (see Fact 5). 

A right simplex whose tree of legs is a path is called a Schlaefli simplex. 

The face of a right simplex spanned by all pendent vertices (i.e., vertices of degree one) of the tree of 
legs is called the hypotenuse of the simplex. 

A net on a simplex is a subset of the set of edges (not necessarily connected) such that every vertex of 
the simplex belongs to some edge in the net. A net is metric if for each edge in the net its length is given. 

A box in a Euclidean n-space is a parallelepiped all of whose edges at some vertex (and then at all 
vertices) are mutually perpendicular. 


Facts (cf. [Fie57]): 


1. A simplex is hyperacute if and only if it has no blue edge in its coloring. 

2. The set of red edges connects all the vertices of the simplex. 

3. Ifwe color the edges of an n-simplex by the two colors red and blue in such a way that the red edges 
connect all vertices, then there exists such deformation of the simplex that opposite red edges there 
are acute, opposite blue edges obtuse, and opposite uncolored edges right interior angles. 
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Every n-simplex has at least 1 acute interior angles. 

There exist right n-simplexes. The red edges span a tree containing all the vertices of the simplex. 
The legs of a right simplex are mutually perpendicular. 

The tree of legs of a right n-simplex can be completed to a (n-dimensional) box; its center of 
symmetry is thus the circumcenter of the simplex. Conversely, every right n-simplex can be obtained 
by choosing among the edges of some box a subset of n edges with the property that any two are 
perpendicular and together form a connected set. These are then the legs of the simplex. 

Let Gr = (N, E, W) bea tree with the numbered vertex set N = {1,2,...,1+ 1} and edge set E; 
let every edge {i, j} € E be assigned a positive number w;;. Construct the matrices Qo = [4,5], 
r,s =0,1,...,n +1, and M = [m,j], i,j = 1,...,n+ 1,as follows: 


1 . 
qo = qoi = qi =d;—-2, i€N, 


ijeE 1) 


where dj is the degree of the vertex i in Gr, 


qij = Wij if {i, j} EE, gi= De, Wijp> Gij = 0 otherwise; 
j(i,j EE 


. -1 -1 
m;,=0, ié€N, Mij = Wig, + Weg boot We 


where i,j € N,i # j,andi,k,,hky,...,k,, 7 is the (unique) path in Gy from i to j. 
0 T 
Then the matrices Qy and My = -— satisfy 


Mo Qo = —2In+2 


and they are matrices corresponding to a right n-simplex whose tree of legs is Gr. 
The inhomogeneous barycentric coordinates of the circumcenter of a right n-simplex are qo; = 
1 — }d;, where d; is the degree of the vertex i in the tree of legs. 


. The hypotenuse of a Schlaefli simplex is the longest edge; the midpoint of the hypotenuse is the 


circumcenter of the simplex. 


. Every face of a Schlaefli simplex is also a Schlaefli simplex. 
. The Schlaefli simplex is the only simplex all 2-dimensional faces of which are right triangles. 
. An n-simplex © is a Schlaefli simplex if and only if there exist real distinct numbers c),...,¢n41 


such that the Menger matrix M = [mjj;] of & has the form mj; = |c; — c;|. If this holds for 
Cy > C2 > +++ > Cy41, then, in the usual notation, the edges {A;y, Axii}, k = 1,...,n, are 
the legs of X. 

Let & be a hyperacute n-simplex. Then the Gramian Q of & is a singular M-matrix of rank n, with 
annihilating vector 1. 

Every face of a hyperacute simplex & is also a hyperacute simplex. The distribution of acute and 
right angles in the face is completely determined by the distribution of acute and right angles in 
the simplex as follows: If the face is determined by vertices with indices in S, the edge {A;, Aj} 
in the face will be red if and only if one can proceed from the vertex A; to the vertex A; along a 
path in the set of red edges of & which does not contain any vertex with index in S (except A; 
and Aj). 

([Fie61]). In the extended colored graph of an n-simplex, n > 2, the red part has vertex connectivity 
at least two. 

([Fie61]). In the definition of the extended graph and extended coloring, the vertex n + 2 has no 
privileged position in the following sense: Let G be the extended colored graph of some n-simplex 
), with vertex n + 2 corresponding to the circumcenter of by. If k € {1,2,...,n + 1}, then 
there exists another n-simplex X2 whose extended colored graph is G and such that k is the vertex 
corresponding to the circumcenter of Xp. 
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FIGURE 66.1 Coloring of triangles. 


18. Every face (of dimension at least two) of a totally hyperacute simplex is also a totally hyperacute 
simplex. The extended coloring of the face is by the extended coloring of the simplex uniquely 
determined; it is obtained in the same way as in the usual coloring ofa hyperacute simplex in Fact 15. 

19. Let X be a totally hyperacute n-simplex. Then the extended graph of & is either a cycle, and then 
& is a Schlaefli simplex, or it has vertex connectivity at least three. 

20. ([Fie61]) Let a metric net N on a simplex be given. There exists a simplex of maximum volume 
with this net if and only if the net N is connected, i.e., if it is possible to pass from any vertex of the 
net to any other vertex using edges in the net only. In addition, every simplex with this maximum 
volume has the property that every interior angle opposite an unspecified edge of the simplex 
(i.e., not belonging to N) is right. 


Examples: 


1. Figure 66.1 shows examples of three colored triangles. In these diagrams, the heaviest line represents 
red, the ordinary line represents blue, and the light gray line represents white. 

2. Figure 66.2 shows examples of the two types of right tetrahedra. The dashed lines indicate the box 
described in Fact 7. The tetrahedron on the right is a Schlaefli simplex. 

3. Figure 66.3 shows examples of extended graphs of triangles in Figure 66.1. 

4. Weapply the result in Fact 20 to so-called cyclic simplexes (cf.[Fie61]), which are maximum volume 
simplexes in the case that the metric net is a cycle. We call a simplex regularly cyclic if all edges in 
this net have the same length. 

The Gramian of the regularly cyclic n-simplex is a multiple of the matrix 


2 -1 O «:- 0 -il 
—l 2 -l ::: 0 0 
0 0 0 2 -l1 
—l 0 O «+. -1 2 
with n + 1 rows and columns, if the net is formed by the edges {Ay, Agi}, k = 1,...,n, and 


{An41, Ai}. 

The corresponding Menger matrix is then proportional to the matrix with entries mj = 
|i — k|(n+ 1 — |i — k]). It is possible to show that any two of the edges in the cycle span the 
same angle. 


FIGURE 66.2 The two types of right tetrahedra. 
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FIGURE 66.3 Extended graphs of triangles. 


It is immediate that the regularly cyclic 2-simplex is the equilateral triangle. The regularly cyclic 
3-simplex is the tetrahedron, which is obtained from a square by parallelly lifting one diagonal in 
the perpendicular direction to the plane so that the distance of the two new diagonals equals half 
of the length of each diagonal. 

Thus, the volume of every n-simplex with vertices Ay, Aj,...,A,+) in a Euclidean n-space for 
which all the edges {A;, Ap}, {Az, As}, ...5 {Ans Anai}, {An+1, Ai} have length one, does not exceed 


1 / (tn 
nl n® . 


66.5 An Application to Resistive Electrical Networks 


We conclude with applications of matrices and graphs in another, maybe surprising, field. 


Definitions: 


A resistive electrical network is a network consisting of a finite number of nodes, say 1,2,...,, some 
pairs of which are directly connected by a conductor of some resistance. The assumptions are that the whole 
network is connected, and that there are no other electrical elements than resistors. As usual, conductivity 
of the conductor between two nodes is the reciprocal of the resistance of that conductor. If there is no 
direct connection between a pair of nodes, we set the conductivity of that “conductor” as zero. 

A resistive electrical network of which just some of the nodes, outlets, are accessible, is called a black-box. 
The matrix of mutual resistances between the outlets of a black-box is called a black-box matrix. 

The problem is: Characterize the set of all possible black-box n x n matrices. 


Facts ({Moo68], [Fie78]): 


1. Resistances of conductors in series add and conductivities of conductors in parallel add. If conduc- 
tors having resistances Rj, Rp are placed in series (see left illustration in Figure 66.4) the resistance 
R between the nodes is R; + Rp). If conductors having resistances Rj, Ro are placed in parallel 
(see right illustration in Figure 66.4) the resistance R between the nodes satisfies z= g@ + %- 

2. The 3 x 3 black-box matrices are all matrices 


Ro 


FIGURE 66.4 Resistors in series and parallel. 
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in which the numbers rj, 713, and r23 are nonnegative, at least two different from zero, and fulfill 
the nonstrict triangle inequality. 

3. Then x n black-box matrices are exactly Menger matrices of hyperacute (n — 1)-simplexes. 

4. Ifa black-box matrix is given, then the corresponding network can be realized as such for which 
the conductivities between pairs of outlets are equal to the negatively taken corresponding entries 
of the Gramian of the simplex whose Menger matrix the given matrix is. 

5. ([Fie78]) Let the given black-box B with n outlets correspond to an (n — 1)-simplex X. Let S be 
a nonvoid proper subset of the set of outlets of B. Join all outlets in S by shortcuts. The resulting 
device can be considered as a new black-box Bs, which has just one outlet instead those outlets in 
S and, of course, all the remaining outlets. We construct the simplex corresponding to Bs and its 
black-box matrix as follows. 

If L is the linear space in the corresponding E"~' determined by the vertices corresponding to S, 
project orthogonally all the remaining vertices as well as L itself on (some) orthogonal complement 
L+ to L, thus obtaining a new simplex. The Menger matrix Ms of this simplex will be the black-box 
matrix of Bs. An algebraic construction is to pass from the Menger matrix of &, to the Gramian 
Q of & using Fact 8 of section 66.3; in this Gramian Q, add together all the rows corresponding 
to S into a single row and all the columns corresponding to S into a single column. The resulting 
symmetric matrix Q is againa singular M-matrix with row-sums zero. The simplex whose Gramian 
is Q is then that whose Menger matrix is Bs. 

6. Let S; and S) be two disjoint nonempty subsets of the set of outlets of a black-box, and let = be the 
simplex in the geometric model. Join all outlets in S; with a source of potential zero, and all outlets 
in Sy with the source of potential one. What will be the distribution of potentials in the remaining 
outlets using the geometric interpretation? 

The answer is: Let L, L2, respectively, be linear spaces determined by vertices of © correspond- 
ing to S), So, respectively. There exists a unique pair of parallel hyperplanes H, and Hp, such that 
H, contains L; and H) contains Ly, and the distance between H, and H) is maximal among all 
such pairs of parallel hyperplanes. Then the square of the distance of the hyperplanes H, and H) 
measures the resistance between S,; and S> (similarly like the square of the distance between two 
vertices of © measures the resistance between the corresponding outlets), and if Vo is a vertex 
corresponding to an outlet So, then the potential in Sp is obtained by linear interpolation corre- 
sponding to the position of the hyperplane Hp containing Vo and parallel to H with respect to the 
hyperplanes H; and H). Let us remark that the whole simplex © is in the layer between H; and Hp, 
thanks to the property of hyperacuteness of D. 


Examples: 


1. Suppose the three nodes 1, 2, and 3 are connected as follows: The nodes 1 and 2 are connected by 
a conductor of resistance 18 ohms, the nodes 1 and 3 by conductor of resistance 12 ohms, and the 
nodes 2 and 3 by conductor of resistance 6 ohms. We compute the resistances for the black-box 
with the three outlets 1, 2, and 3. For rj, the total resistance between outlets 1 and 2, note that 
there are two parallel paths between 1 and 2: the direct path with resistance 18 ohms, and the path 


through node 3 with resistance 6 ohms + 12 ohms = 18 ohms. Thus = = + + 7 = z, $0 112 = 
9 ohms. Similar computations show that the resistance between outlets 1 and 3 is 8 ohms and the 
0 9 | 
resistance between outlets 2 and 3 is 5 ohms. The black-box matrix is s 0 i : 
8 5 0 


2. Let us remark that the properties of the outlets of a black-box do not depend on the way the 
conductors and resistors in the box are set. Here is another way the black-box-matrix in Example 1 
could be obtained: There are no direct connections between nodes 1, 2, and 3, but there is a 
fourth node 4, which is connected with node 1 by a conductor of resistance 6 ohms, with node 2 
by a conductor of resistance 3 ohms, and with node 3 by a conductor of resistance 2 ohms. Since 
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resistors in series add, this produces the same black-box matrix. If we then connect the outlets 1 
and 2 bya short-circuit (conductor of no resistance), in both cases the resulting resistance between 
the new outlet {1,2} and outlet 3 will be the same, equal to 4 ohms. 
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The topics of this chapter and the next (on group representations) are closely related. Here we consider 
some particular groups that arise most naturally as matrix groups or quotients of them, and special 
properties of matrix groups that are not shared by arbitrary groups. In representation theory, we consider 
what we learn about a group by considering all its homomorphisms to matrix groups. In this chapter 
we discuss properties of specific matrix groups, especially the general linear group (consisting of all 
invertible matrices of given size over a given field) and the related “classical groups.” Most group theoretic 
terminology is standard and can be found in any textbook or in the Preliminaries in the Front Matter of 
the book. 


67.1 Introduction 


Definitions: 


The general linear group GL(n, F ) is the group consisting of all invertible n x n matrices over the field F. 

A matrix group is a a subgroup of GL(n, F ) for some natural number n and field F. 

If V is a vector space of dimension n over F, the group of invertible linear operators on V is denoted 
by GL(V). 

A linear group of degree n is a subgroup of GL(V) (where dim V = n). (A subgroup of GL(n, F) is a 
linear group of degree n, since ann x n matrix can be viewed as a linear operator acting on F” by matrix 
multiplication.) 

A linear group G < GL(V) is said to be reducible if there is a G-invariant subspace U of V other than 
{0} and V, and is irreducible otherwise. If G < GL(V) is irreducible, then V is called G-irreducible. 

A linear group G < GL(V) is said to be decomposable if V is the direct sum of two nonzero G -invariant 
subspaces, and is indecomposable otherwise. 

If V can be expressed as the direct sum of G-irreducible subspaces, then G is completely reducible. 
(An irreducible group is completely reducible.) 

If the matrix group G < GL(n, F) is irreducible regarded as a subgroup of GL(n, K ) for any algebraic 
extension K of F, we say that G is absolutely irreducible. 

A linear group of degree n is unipotent if all its elements have n eigenvalues equal to 1. 

Let X and Y be group-theoretic properties. 

A group G is locally X if every finite subset of G is contained in a subgroup with property X. 
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Facts: 


Handbook of Linear Algebra 


For these facts and general background reading see [Dix71], [Sup76], and [Weh73]. 


1. 
2. 
a 


11. 


12. 


13; 


14. 


15. 


If V is a vector space of dimension n over F,, then GL(V) is isomorphic to GL(n, F). 

Every finite group is isomorphic to a matrix group. 

Basic facts from linear algebra about similarity of matrices can be interpreted as statements about 

conjugacy classes in GL(n, F ). For example: 

* Two nonsingular matrices are conjugate in GL(n, F ) if and only if they have the same invariant 
factors. 


If F is algebraically closed, then two nonsingular matrices are conjugate in GL(n, F) if and only 

if they have the same Jordan canonical form. 

* Two real symmetric matrices are conjugate in GL(n, R) if and only if they have the same rank 
and signature. 

(See Chapter 6 for more information on the Jordan canonical form and invariant factors and 

Chapter 12 for more information on signature.) 


. Amatrix group G of degree n is reducible if and only if there exists a nonsingular matrix M € F"*" 


By 


and k with 1 < k < n—1 such that for all A € G, M7! AM is of the form & B 
22 


, where 


By E FExk Boy € BF (n-k)x(n-k) | 


. (See Chapter 68) The image of a representation of a group is a linear group. The image of a matrix 


representation of a group is a matrix group. We apply descriptions of the linear group to the 
representation: If p : G > GL(V) is a representation, and p(G) is irreducible, indecomposable, 
absolutely irreducible, etc., then we say that the representation p is irreducible, etc. 


. If every finitely generated subgroup of a group G is isomorphic to a linear group of degree n over 


a field F (of arbitrary characteristic), then G is isomorphic to a linear group of degree n. 


. Any free group is linear of degree 2 in every characteristic. More generally, a free product of linear 


groups is linear. 


. (Maschke’s Theorem) Let G be a finite linear group over F, and suppose that the characteristic of 


F is either zero or coprime to |G|. If G is reducible, then it is decomposable. 


. Alocally finite linear group in characteristic zero is completely reducible. 
10. 


(Clifford’s Theorem) Let G be an irreducible linear group on a vector space V of dimension n, 
and let N be a normal subgroup of G. Then V is a direct sum of minimal N-spaces W,,..., Wa 
permuted transitively by G. In particular, d divides n, the group N is completely reducible, and the 
linear groups induced on W; by N are all isomorphic. 

A normal (or even a subnormal) subgroup of a completely reducible linear group is completely 
reducible. 

A unipotent matrix group is conjugate (in the general linear group) to the group of upper unit 
triangular matrices. 

A linear group G on V has a unipotent normal subgroup U such that G/U is isomorphic to a 
completely reducible linear group on V; the subgroup U isa nilpotent group of class at most n — 1, 
where n = dim(V). 

(Mal’cev) If every finitely generated subgroup of the linear group G is completely reducible, then 
G is completely reducible. 

Let G be a linear group on F", where F is algebraically closed. Then G is irreducible if and only if 
the elements of G span the space F”*" of all n x n matrices over F. 


Examples: 


1. 


The matrix group 


Whe Wl 
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is decomposable and completely reducible: the subspaces of R? spanned by the vectors [1, 1] and 


1 -l 
[—1,2]? are G-invariant. That is, with M = ; ; } for any A € G, M7! AM is a diagonal 


(eco 


is reducible, but neither decomposable nor completely reducible. 


3. The matrix group 
cosx  sinx 
: :xER 
—sinx cosx 


of real rotations is irreducible over R but not over C: The subspace spanned by [1,i]7 is invariant. 


matrix. 
2. The matrix group 


The matrices in this group span the 2-dimensional subspace of R?”? consisting of matrices A = [a; i] 
satisfying the equations a), = az) and a) + ay = 0. 
4. A group is locally finite if and only if every finitely generated subgroup is finite. 


67.2 The General and Special Linear Groups 


Definitions: 


The special linear group SL(n, F ) is the subgroup of GL(n, F ) consisting of matrices of determinant 1. If 
V is a vector space of dimension n over F , the group of invertible linear operators on V having determinant 
1 is denoted by SL(V). 

The special linear group SL(n, F ) is the subgroup of GL(n, F ) consisting of matrices of determinant 1. 

The projective general linear group and projective special linear group PGL(n, F) and PSL(n, F ) are 
the quotients of GL(n, F ) and SL(n, F ) by their normal subgroups Z and ZNSL(n, F ), respectively, where 
Z is the group of nonzero scalar matrices. 

Notation: If F = GF(q) is the finite field of order q, then GL(n, F) is denoted GL(n, q), SL(n, F) is 
denoted SL(n, q), etc. 

A transvection is a linear operator T on V with all eigenvalues equal to 1 and satisfying rank(T —I) = 1. 


Facts: 
For these facts and general background reading see [HO89], [Tay92], or [Kra02]. 


1. The special linear group SL(n, F ) is a normal subgroup of the general linear group GL(n, F ). 
2. The order of GL(n, q) is equal to the number of ordered bases of GF(q)", namely 


n-1 n-1 
|GL(n, 4) =] ][(@" -4') = 4"? [qr - v. 
i=0 i=0 


3. A transvection has determinant 1, and so lies in SL(V). 

4. A transvection on F” has the form I + vw" for some v,w € F” andw’v = 0, and anything in this 
form is a transvection. 

5. A transvection T on V has the form T : x > x+ f(x)v, where v € V, f € V*, and f(v) = 0, 
and anything in this form is a transvection. 

6. The group SL(n, F) is generated by transvections, for any n > 2 and any field F. 
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7. The group PSL(n, F) is simple for all n > 2 and all fields F , except for the two cases PSL(2, 2) and 
PSL(2, 3). The groups PSL(2, 2) and PSL(2, 3) are isomorphic to the symmetric group on 3 letters 
and the alternating group on 4 letters, respectively. The groups PSL(2, 4) and PSL(2, 5) are both 
isomorphic to the alternating group on 5 letters. 


Examples: 
1. GL(2, 2) = SL(2, 2) = PSL(2, 2) = 


2. SL(2,3) = 


28 20 Se 


is a transvection. T = I + vw! with v = [—2,2,4]", and w = [7,5,1]". 


67.3. The BN Structure of the General Linear Group 


Definitions: 
A BN-pair (or Tits system) is an ordered quadruple (G, B, N, S) where 
* G isa group generated by subgroups B and N. 
¢ T:= BM Nis normal in N. 
* Sisasubset of W := N/T and S generates W. 
¢ The elements of S are all of order 2. 
* Ifp,o € NandpT € S,thenpBo C BoBU BooB. 
* IfoT € S, then pBp S$ B. 


If (G, B, N, S) isa BN-pair, the subgroups B and W = N/T are known as the Borel subgroup and Weyl 
group of G. 

A parabolic subgroup of G (relative to a given BN-pair) is a subgroup of the form P; = (B,s;:i € I) 
for some subset I of {1,...,|S|}. 


Facts: 
For these facts and general background reading see [HO89], [Tay92], or [Kra02]. 
1. The general linear group GL(n, F ) with n > 2 has the following Tits system: 
* B is the group of upper-triangular matrices in G. 


* U the group of unit upper-triangular matrices (with diagonal entries 1). 
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* T the group of diagonal matrices. 

* N is the group of matrices having a unique nonzero element in each row or column. 

° § ={s;:i =1,...,n— 1}, where s; = P;T and P; is the reflection which interchanges the ith 
and (i + 1)st standard basis vectors (P; is obtained from the identity matrix by interchanging 
rows i andi + 1). 

¢ N is the normalizer of T in GL(n, F). 

°° B=UT. 

*BON=T. 

* N/T is isomorphic to the symmetric group S,. 

2. If G has a BN-pair, any subgroup of G containing B is a parabolic subgroup. 

3. In GL(n, F), there are 2"~! parabolic subgroups for the BN-pair in Fact 1, hence, there are Dia 
subgroups of GL(n, F ) containing the subgroup B of upper-triangular matrices. 

4. More generally, with respect to any basis of V there is a BN-structure. The terms Borel subgroup 
and parabolic subgroup are used to refer to the subgroups defined with respect to an arbitrary basis. 
All the Borel subgroups of GL(V) are conjugate. The maximal parabolic subgroups are precisely 
the maximal reducible subgroups. 


Examples: 
1. The maximal parabolic subgroups of GL(n, F) with the BN-pair in Fact 1 are those for which 
I = {1,...,2 — 1} \ {k} for some k; it is easy to see that in this case P; is the stabilizer of the 
subspace spanned by the first k basis vectors. This subgroup consists of all matrices with block form 
as in 67.1, Fact 4. 


67.4 Classical Groups 


The classical groups form several important families of linear groups. We give a brief description here, and 
refer to the books [HO89], [Tay92], or the article [Kra02] for more details. For information on bilinear, 
sesquilinear, and quadratic forms, see Chapter 12. 


Definitions: 


A g-sesquilinear form B is g-Hermitian if B(v,w) = y(B(w,v)) for all v,w € V. In the case where 
F = Cand 9g is conjugation, a g-Hermitian form is called a Hermitian form. 

A formed space is a finite dimensional vector space carrying a nondegenerate y-Hermitian, symmetric, 
or alternating form B. 

A classical group over a formed space V is the subgroup of GL(V) consisting of the linear operators 
that preserve the form. We distinguish three types of classical groups: 


1. Orthogonal group: Preserving a nondegenerate symmetric bilinear form B. 
2. Symplectic group: Preserving a nondegenerate alternating bilinear form B. 
3. Unitary group: Preserving a nondegenerate o -Hermitian form B, witho F 1. 


We denote a classical subgroup of GL(V) by O(V), Sp(V), or U(V) depending on type. If necessary, 
we add extra notation to specify which particular form is being used. If V = F", we also write O(n, F), 
Sp(n, F), or U(n, F). 

The Witt index of a formed space V is the dimension of the largest subspace on which the form is 
identically zero. The Witt index of the corresponding classical group is the Witt index of the formed space. 

An isometry between subspaces of a formed space is a linear transformation preserving the value of the 
form. 

A representation of a group G over the complex numbers is said to be unitary if its image is contained 
in the unitary group. 
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Facts: 


Handbook of Linear Algebra 


For these facts and for general background reading see [HO89], [Tay92], or [Kra02]. 


1. 


10. 


11. 


The only automorphism of R is the identity, so any sesquilinear form ona real vector space is bilinear 
and any g-Hermitian form is symmetric. A real formed space has a symmetric or alternating bilinear 
form as its form. The classical subgroups of a real formed space are orthogonal or symplectic. 
The only automorphisms of C that preserve the reals are the identity and complex conjugation. 
Any g-Hermitian form such that ¢ preserves the reals is a Hermitian form. 

Classification of classical groups up to conjugacy in GL(n, F ) is equivalent to classification of forms 

of the appropriate type up to the natural action of the general linear group together with scalar 

multiplication. Often this is a very difficult problem; the next fact gives a few cases where the 
classification is more straightforward. 

(a) A nondegenerate alternating form on V = F” exists if and only if n is even, and all such forms 
are equivalent. So there is a unique conjugacy class of symplectic groups in GL(n, F) if n is 
even (with Witt index n/2), and none if n is odd. 

(b) Let F = GF(q). Then, up to conjugacy, GL(n,q) contains one conjugacy class of unitary 
subgroups (with Witt index [1/2] ), one class of orthogonal subgroups if n is odd (with Witt 
index (n — 1)/2), and two classes if n is even (with Witt indices n/2 and n/2 — 1). 

(c) Anondegenerate symmetric bilinear form on R” is determined up to the action of GL(n, R) by 
its signature. Its Witt index is min{s, t}, where s and ¢ are the numbers of positive and negative 
eigenvalues. So there are [n/2] + 1 conjugacy classes of orthogonal subgroups of GL(n, R), 
with Witt indices 0,1,...,|n/2]. 

(Witt’s Lemma) Suppose that U; and U; are subspaces of the formed space V, and h : U; > U} is 

an isometry. Then there is an isometry g of V that extends h. 

From Witt’s Lemma it is possible to write down formulas for the orders of the classical groups over 

finite fields similar to the formula in Fact 2 general linear group. 

The analogues of Facts 6 and 7 in section 67.2 hold for the classical groups with nonzero Witt index. 

However, the situation is more complicated. Any symplectic transformation has determinant 1, 

so Sp(2r, F) < SL(2r, F). Moreover, Sp(2r, F) is generated by symplectic transvections (those 

preserving the alternating form) for r > 2, except for Sp(4, 2). Similarly, the special unitary group 

SU(n, F ) (the intersection of U(n, F ) with SL(n, F )) with positive Witt index is generated by unitary 

transvections (those preserving the Hermitian form), except for SU(3, 2). Results for orthogonal 

groups are more difficult. See [HO89] for more information. 


. Like the general linear groups, the classical groups contain BN-pairs (configurations of subgroups 


satisfying conditions like those in the previous section). The difference is that the Weyl group 
W = N/H is not the symmetric group, but one of the other types of Coxeter group (finite groups 
generated by reflections). 

Although this treatment of classical groups has been as far as possible independent of fields, for most 
of mathematics, the classical groups over the real and complex numbers are the most important, 
and among these, the real orthogonal and complex unitary groups preserving positive definite 
forms most important; see [Wey39]. 

The theory can be extended to classical groups over rings. This has important connections with 
algebraic K-theory. The book [HO89] gives details. 

Every representation of a finite group is equivalent to a unitary representation. 


Examples: 


ie 


2: 


The function B given by B((x1, y1), (x2, y2)) = X1¥2 — X21 is an alternating bilinear form on F?. 
Any matrix with determinant 1 will preserve this form. So Sp(2, F) = SL(2, F). 

The symmetric group Ss acts on F°, where F is the field with two elements. It preserves the 
1-dimensional subspace U spanned by (1, 1, 1, 1, 1, 1), as well as the 5-dimensional subspace 
consisting of vectors with coordinate sum zero. The usual dot product on F° is alternating when 
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restricted to W, and its radical is U, so it induces a symplectic form on W/U. Thus S¢ is a subgroup 
of the symplectic group Sp(4, 2). Since both groups have order 720, we see that Sp(4,2) = S¢. 
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Representation theory is the study of the various ways a given group can be mapped into a general linear 
group. This information has proven to be effective at providing insight into the structure of the given 
group as well as the objects on which the group acts. Most notable is the central contribution made by 
representation theory to the complete classification of finite simple groups [Gor94]. (See also Fact 3 of 
Section 68.1 and Fact 5 of Section 68.6.) 

Representations of finite groups can be defined over an arbitrary field and such have been studied 
extensively. Here, however, we discuss only the most widely used, classical theory of representations over 
the field of complex numbers (many results of which fail to hold over other fields). 


68.1 Basic Concepts 


Throughout, G denotes a finite group, e denotes its identity element, and V denotes a finite dimensional 
complex vector space. 


Definitions: 


The general linear group of a vector space V is the group GL(V) of linear isomorphisms of V onto itself 
with operation given by function composition. 

A (linear) representation of the finite group G (over the complex field C) is a homomorphism 
p = py :G > GL(V), where V is a finite dimensional vector space over C. 

The degree of a representation py is the dimension of the vector space V. 

Two representations p : G > GL(V) and p’: G > GL(V’) are equivalent (or isomorphic) if there 
exists a linear isomorphism t: V > V’ such that t 0 p(s) = p’(s) oT foralls € G. 

Given a representation py of G, a subspace W of V is G-stable (or G-invariant) if py(s)(W) C W for 
alls € G. 

If py is a representation of G and W is a G-stable subspace of V, then the induced maps pw:G —> 
GL(W) and pyjw:G — GL(V/W) are the corresponding subrepresentation and quotient represen- 
tation, respectively. 

A representation py of G with V ¥ {0} is irreducible if V and {0} are the only G-stable subspaces of 
V; otherwise, py is reducible. 
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kernel of a representation py of G is the set of all s € G for which py(s) = ly. 


A representation of G is faithful if its kernel consists of the identity element alone. 
An action of G ona set X is a function G x X — X, (s,x) t> sx, satisfying 


(st)x = s(tx) foralls,t € Gandx € X, 
ex =x forallx € X. 


A CG-module is a finite-dimensional vector space V over C together with an action (s,v) > sv of G 


on V that is linear in the variable v, meaning 


s(v+w)=sv-+sw foralls € Gandv,w € V, 
s(av) =a(sv) foralls € G,v € Vanda eC. 


(See Fact 6 below.) 


Facts: 


The following facts can be found in [Isa94, pp. 4-10] or [Ser77, pp. 3-13, 47]. 


1. If op = py isa representation of G, then 
* ple) =1y, 

* o(st) = p(s)o(t) for alls,t € G, 
* p(s—!) = p(s)! foralls EG. 

2. A representation of G of degree one is a group homomorphism from G into the group C* of 
nonzero complex numbers under multiplication (identifying C* with GL(C)). Every representa- 
tion of degree one is irreducible. 

3. The group G is abelian if and only if every irreducible representation of G is of degree one. 

4. Maschke’s Theorem: If py is a representation of G and W is a G-stable subspace of V, then there 
exists a G-stable vector space complement of W in V. 

5. Schur’s Lemma: Let p:G — GL(V) and p’:G > GL(V’) be two irreducible representations of 
G and let f : V > V’ bea linear map satisfying f o p(s) = p'(s) o f foralls € G. 

* If p’ is not equivalent to p, then f is the zero map. 
¢ If V’ = Vand p’ = p, then f isa scalar multiple of the identity map: f = aly forsomea € C. 

6. Ife = pv isarepresentation of G, then V becomes aC G-module with action given bysv = p(s)(v) 
(s € G,v € V). Conversely, if V is a CG-module, then py(s)(v) = sv defines a representation 
py:G — GL(V) (called the representation of G afforded by V). The study of representations of 
the finite group G is the same as the study of CG-modules. 

7. The vector space CG over C with basis G is a ring (the group ring of G over C) with multiplication 
obtained by linearly extending the operation in G to arbitrary products. If V is a CG-module 
and the action of G on V is extended linearly to a map CG x V — V, then V becomes a (left 
unitary) CG-module in the ring theoretic sense, that is, V satisfies the usual vector space axioms 
(see Section 1.1) with the scalar field replaced by the ring CG. 

Examples: 


See also examples in the next section. 


1. 


2. 


Let n € Nand let w € C be an nth root of unity (meaning w” = 1). Then the map p : Z, > C* 
given by p(m) = w" is a representation of degree one of the group Z,, of integers modulo n. It is 
irreducible. 

Regular representation: Let V = CG be the complex vector space with basis G. For each s € G 
there is a unique linear map p(s) : V — V satisfying p(s)(t) = st for allt € G. Thenp: G > 
GL(V) isa representation of G called the (left) regular representation. If |G| > 1, then the regular 
representation is reducible (see Example 3 of Section 68.5) . 
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3. Permutation representation: Let X bea finite set, let (s, x) H sx bean action of G on X, and let V be 
the complex vector space with basis X. For each s € G there is a unique linear map p(s): V > V 
satisfying p(s)(x) = sx for all x € X. Then p : G > GL(V) isa representation of G called 
a permutation representation. The regular representation of G (Example 2) is the permutation 
representation corresponding to the action of G on itself given by left multiplication. 

4. The representation of G of degree 1 given by p(s) = 1 € C% for alls € G is the trivial 
representation. 

5. Direct sum:If V and W are CG-modules, then the C-vector space direct sum V @ W isa CG-module 
with action given by s(v,w) = (sv,sw) (s € G,v e Viwe W). 

6. Tensor product: If V; is a CG,-module and V, is a CG2-module, then the C-vector space tensor 
product V, ® V2 isa C(G; x G2)-module with action given by (sj), s2)(v1 ® v2) = (svi) ® (s2¥2) 
(s; € Gi,v; € V;). Ifboth groups G; and G2 equal the same group G, then V, ® V2 isa CG-module 
with action given by s(v; ® v2) = (sv1) @ (sv2) (s € G, vj € Vj). 

7. Contragredient: If V isa CG-module, then the C-vector space dual V* is a CG-module (called the 
contragredient of V) with action given by (sf)(v) = f(s~!v) (s € G, fe V*,v eV). 


68.2 Matrix Representations 


Throughout, G denotes a finite group, e denotes its identity element, and V denotes a finite dimensional 
complex vector space. 


Definitions: 


A matrix representation of G of degree n (over the field C) is a homomorphism R:G — GL,(C), 
where GL,,(C) is the group of nonsingular n x n matrices over the field C. (For the relationship between 
representations and matrix representations, see the facts below.) 

The empty matrix is a 0 x 0 matrix having no entries. The trace of the empty matrix is 0. GLo(C) is 
the trivial group whose only element is the empty matrix. 

Two matrix representations R and R’ are equivalent (or isomorphic) if they have the same degree, say 
n, and there exists a nonsingular n x n matrix P such that R'(s) = PR(s)P™! foralls € G. 

A matrix representation of G is reducible if it is equivalent to a matrix representation R having the 
property that for each s € G, the matrix R(s) has the block form 


oe es “| 
0 Y(s) 


(block sizes independent of s). 
A matrix representation is irreducible if it has nonzero degree and it is not reducible. 
The kernel of a matrix representation R of G of degree n is the set of alls € G for which R(s) = I. 
A matrix representation of G is faithful if its kernel consists of the identity element alone. 


Facts: 
The following facts can be found in [Isa94, pp. 10-11, 32] or [Ser77, pp. 11-14]. 


1. If R isa matrix representation of G, then 
* R(e)=T, 
¢ R(st) = R(s)R(t) for alls,t € G, 
* R(s~!) = R(s)7! foralls eG. 
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2. Ife = py isarepresentation of G of degree n and B isan ordered basis for V, then Ryp,g(s) = [p(s] 
defines a matrix representation Rpg: G — GL, (C) called the matrixrepresentation of G afforded 
by the representation p (or by the CG-module V) with respect to the basis 5. Conversely, if R 
is a matrix representation of G of degree n and V = C”, then p(s)(v) = R(s)v (s € G,v € V) 
defines a representation p of G and R = R,,z, where B is the standard ordered basis of V. 

3. If R and R’ are matrix representations afforded by representations p and p’, respectively, then R 
and R’ are equivalent if and only if p and p’ are equivalent. In particular, two matrix representations 
that are afforded by the same representation are equivalent regardless of the chosen bases. 

4. If op = py isa representation of G and W isa G-stable subspace of V and a basis for W is extended 
to a basis B of V, then for each s € G the matrix R,,~(s) is of block form 


X(s) ae 


Rp,b(s) = 
(0) Y(s) 


where X and Y are the matrix representations afforded by pw (with respect to the given basis) and 
Pvw (with respect to the induced basis), respectively. 

5. If the matrix representation R of G is afforded by a representation p, then R is irreducible if and 
only if ¢ is irreducible. 

6. The group G is Abelian if and only if every irreducible matrix representation of G is of degree one. 

7. Maschke’s Theorem (for matrix representations): If R is a matrix representation of G and for each 
s € G the matrix R(s) is of block form 


X(s) Z(s) 
R(s) = 
0 Y(s) 
(block sizes independent of s), then R is equivalent to the matrix representation R’ given by 
X(s 0 
Rs) -| a | 
0 Y(s) 


(s €G). 

8. Schur relations: Let R and R’ be irreducible matrix representations of G of degrees n and n’, 
respectively. For 1 < i,j < nand1 < i’, j’ < n’ define functions Tips l jj :G—> Cby R(s) = 
[rij(s)], R'(s) = [rj ;-(s)] (s € G). 


* If R’ is not equivalent to R, then for all 1 <i, j <nand1 <7’, j’ <n’ 


ye Ee) = 0. 


seEG 


* Foralll <i, j,k, <n 


So rij(s"ru(s) = 


sEG 


a ifi =landj =k 


0 otherwise. 


Examples: 


1. An example of a degree two matrix representation of the symmetric group S3 is given by 


re)= | i roa) = |? aE roa) =| ral 


1 0 0 1 =) 
ra)=| 1, ee a2) =| °, el as) =| 7 al 


(Cf. Example 2 of section 68.4.) 
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2. The matrix representation R of the additive group Z; of integers modulo 3 afforded by the regular 
representation with respect to the basis Z; = {0, 1, 2} (ordered as indicated) is given by 


1 0 0 0 0 1 0 
RO)=]0 1 0], ROJY=]1 0 OF], RQ=]0 
0 0 1 0 1 £0 1 


3. Letp: G > GL(V) and p’: G > GL(V’) be two representations of G, let B and B’ be bases of 
V and V’, respectively, and let R = R,p,g and R' = Ry be the afforded matrix representations. 


* The matrix representation afforded by the direct sum V @ V’ with respect to the basis 
{(b, 0), (0, b’) |b € Bb’ € B’} is given bys t R(s) @ R'(s) (direct sum of matrices). 


* The matrix representation afforded by the tensor product V @ V’ with respect to the basis 
{b @b'|b € B,v’ € B’} is given bys + R(s) ® R'(s) (Kronecker product of matrices). 


* The matrix representation afforded by the contragredient V* with respect to the dual basis of B 
is given by s b> (R(s )-!) (inverse transpose of matrix). 


68.3 Characters 


Throughout, G denotes a finite group, e denotes its identity element, and V denotes a finite dimensional 
complex vector space. 


Definitions: 


The character of G afforded by a matrix representation R of G is the function x : G — C defined by 
x(s) =tr R(s). 

The character of G afforded by a representation = py of G is the character afforded by the 
corresponding matrix representation R,,g, where B is a basis for V. 

The character of G afforded by a CG-module V is the character afforded by the corresponding 
representation py. 

An irreducible character is a character afforded by an irreducible representation. 

The degree of a character x of G is the number x (e). 

A linear character is a character of degree one. 

If x; and x» are two characters of G, their sum is defined by (x1 + x2)(s) = xi(s) + X2(s) and their 
product is defined by (x1 x2)(s) = xi(s)xX2(s) (s € G). 

If x is a character of G, its complex conjugate is defined by X(s) = x(s) (s € G), where x (s) denotes 
the conjugate of the complex number x (s). 

The kernel of a character x of G is the set {s € G| x(s) = x(e)}. 

A character x of G is faithful if its kernel consists of the identity element alone. 

The principal character of G is the character 1g satisfying 1g(s) = 1 foralls eG. 

The zero character of G is the character 0g satisfying 0g(s) = 0 for alls € G. 

If x and are two characters of G, then y is called a constituent of x if x = w+ Ww’ with w’ a character 
(possibly zero) of G. 


Facts: 
The following facts can be found in [Isa94, pp. 14-23, 38-40, 59 ] or [Ser77, pp. 10-19, 27, 52]. 


1. The principal character of G is the character afforded by the representation of G that maps every 
s € G to [1] € GL,(C). The zero character of G is the character afforded by the representation of 
G that maps every s € G to the empty matrix in GLo(C). 
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2. The degree of a character of G equals the dimension of V, where py is a representation affording 
the character. 

3. If x isa character of G, then y(s~!) = x(s) and x(t7'st) = x(s) foralls,t € G. 

4. Two characters of G are equal if and only if representations affording them are equivalent. 

5. The number of distinct irreducible characters of G is the same as the number of conjugacy 
classes of G. 

6. Every character x of G can be expressed in the form x = ens mg, where Irr(G) denotes 
the set of irreducible characters of G and where each my is a nonnegative integer (called the 
multiplicity of g as a constituent of x). 

7. A nonzero character of G is irreducible if and only if it is not the sum of two nonzero characters 
of G. 

8. The kernel of a character equals the kernel of a representation affording the character. 

9. The degree of an irreducible character of G divides the order of G. 

10. A character of G is linear if and only if it is a homomorphism from G into the multiplicative group 
of nonzero complex numbers under multiplication. 

11. The group G is abelian if and only if every irreducible character of G is linear. 

12. The sum of the squares of the irreducible character degrees equals the order of G. 

13. Irreducible characters of direct products: Let G, and G; be finite groups. Denoting by Irr(G) the set 
of irreducible characters of the group G, we have Irr(G, x G2) = Irr(G,) x Irr(G2), where an 
element (x1, X2) of the Cartesian product on the right is viewed as a function on the direct product 
Gy X Go via (X15 X2)(S1, 52) = X1(51) X2(52). 

14. Burnside’s Vanishing Theorem: If x is a nonlinear irreducible character of G, then x(s) = 0 for 
somes € G. 


Examples: 


See also examples in the next section. 


1. If V; and V) are CG-modules and x, and x2, respectively, are the characters of G they afford, then 
the direct sum V; ® V> affords the sum x; + x2 and the tensor product V, ® V> affords the product 
X1X2- 

2. If V is a CG-module and x is the character it affords, then the contragredient V* affords the 
complex conjugate character X. 

3. Let X bea finite set on which an action of G is given and let p be the corresponding permutation 
representation of G (see Example 3 of Section 68.1). If x is the character afforded by p, then for each 
s € G, x(s) is the number of ones on the main diagonal of the permutation matrix [p(s )] x, which 
is the same as the number of fixed points of X under the action of s: x(s) = |{x € X|sx = x}I. 
The matrix representation of Z3 given in Example 2 of Section 68.2 is afforded by a permutation 
representation, namely, the regular representation; it affords the character x given by x(0) = 3, 
x(1) = 0, x(2) = 0 in accordance with the statement above. 


68.4 Orthogonality Relations and Character Table 


Throughout, G denotes a finite group, e denotes its identity element, and V denotes a finite dimensional 
complex vector space. 


Definitions: 


A function f :G — C is called a class function if it is constant on the conjugacy classes of G, that is, if 
f(t7'st) = f(s) foralls,t eG. 
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The inner product of two functions f and g from G to C is the complex number 


(fle =D fee). 


sEG 


The character table of the group G is the square array with entry in the ith row and jth column equal 


to the complex number x;(c;), where Irr(G) = {x1,..., xx} is the set of distinct irreducible characters 
of G and {cj,...,c,} is a set consisting of exactly one element from each conjugacy class of G. 
Facts: 


The following facts can be found in [Isa94, pp. 14-21, 30] or [Ser77, pp.10-19]. 


1. Each character of G is a class function. 
2. First Orthogonality Relation: If p and y are two irreducible characters of G, then 


1 1 ifg=w 
(Wc = yo PO) = . sige 


sEG 


3. Second Orthogonality Relation: If s and t are two elements of G, then 


Ss. X05 O= 


xelrr(G) 


{ |G|/c(s) ift is conjugate to s 


0 if t is not conjugate to s, 


where c(s ) denotes the number of elements in the conjugacy class of s. 
4. Generalized Orthogonality Relation: If g and w are two irreducible characters of G and f is an 
element of G, then 
1 a 
— 5° o(st)ws) = 


seEG 


{ g(t)/gle) ifg=y 
0 ifg zw. 


This generalizes the First Orthogonality Relation (Fact 2). 

5. The set of complex-valued functions on G is a complex inner product space with inner product as 
defined above. The set of class functions on G is a subspace. 

6. A character x of G is irreducible if and only if (x, x)g = 1. 

7. The set Irr(G) of irreducible characters of G is an orthonormal basis for the inner product space 
of class functions on G. 

8. If the character x of G is expressed as a sum of irreducible characters (see Fact 6 of Section 68.3), 
then the number of times the irreducible character y appears as asummand is (x, y)g. In particular, 
gy € Irr(G) is a constituent of x if and only if (x, g)g # 0. 

9. Isomorphic groups have identical character tables (up to a reordering of rows and columns). The 
converse of this statement does not hold since, for example, the dihedral group and the quaternion 
group (both of order eight) have the same character table, yet they are not isomorphic. 


Examples: 


1. The character table of the group Z, of integers modulo four is 


0 ol 2 3 


Xo 1 1 1 1 
xi 1 1 —-l -1 
x2 1 -1 1 -!1 
x3 1-1 -l 1 
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2. The character table of the symmetric group S3 is 


|G) (12) _(123) 


Xo 1, 1 
x1 r $4 1 
xX 7 a | | 


Note that x2 is the character afforded by the matrix representation of S3 given in Example 1 of 
Section 68.2. 
3. The character table of the symmetric group Sy, is 


(1) (12) (12)(34) (123) (1234) 
xo he 1 1 1 
XN yo “T 1 1 =] 
xX 2 0 2 = 0 
x3 | | 0 = 
XA 3, Si = 0 1 


4, The character table of the alternating group Ay is 


| G1) (12)(34) (123) (132) 


Xo 1 1 1 1 

XI 1 1 7) wo 

x2 1 1 wo (>) 

x3 3 —1 0 0 
where w = e713 = —} 4 eS 


5. Let py bea representation of G and for each irreducible character y of G put 


g(e) e 
e= TE) S- ls ‘\pv(s):V > V. 
seG 
Then the Generalized Orthogonality Relation (Fact 4) shows that 
T, ifg= 
Tp Ty = i : rae 
0 ifgAy 
and 
gelrr(G) 


where ly denotes the identity operator on V. Moreover, V = Ba 
sum). 


set(@) T,(V) (internal direct 


68.5 Restriction and Induction of Characters 


Throughout, G denotes a finite group, e denotes its identity element, and V denotes a finite dimensional 
complex vector space. 


Definitions: 


If x is a character of G and H isa subgroup of G, then the restriction of x to H is the character x4 of H 
obtained by restricting the domain of x. 
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A character g of a subgroup H of G is extendible to G if g = yx for some character x of G. 
If g is a character of a subgroup H of G, then the induced character from H to G is the character 9° 
of G given by the formula 


1 2 
9°(s)= — Sgt 'st), 
|| teG 


where ¢’° is defined by y°(x) = g(x) ifx € H and y°(x) = Oifx ¢ H. 
If g is a character of a subgroup H of G ands is an element of G, then the conjugate character of g by 
s is the character y’ of HS = s~!Hs given by gy’ (h°) = (h) (h € H), where hS = s~'hs. 


Facts: 


The following facts can be found in [Isa94, pp. 62-63, 73-79] or [Ser77, pp. 55-58]. 


L, 


15. 


The restricted character defined above is indeed a character: x 7 is afforded by the restriction to H 
of a representation affording x. 


. The induced character defined above is indeed a character: Let V be aC H-module affording y and 


put VE = er Vi where G = U,-7 tH (disjoint union) and V; = V for each t. Then VG isa 
CG-module that affords v°, where the action is given as follows: For s € G and v € V,, sv is the 
element hv of V,, where st = t/h (t/ € T,h € H). 


. The conjugate character defined above is indeed a character: y* is afforded by the representation 


of H* obtained by composing the homomorphism H* — H, h* +> h with a representation 
affording 9. 


. Additivity of restriction: Let H bea subgroup of G. If x and x’ are characters of G, then (x¥ + ’)H = 


XH + Xp: 


. Additivity of induction: Let H be a subgroup of G. If y and g’ are characters of H, then (y+ 9')° = 


gf at g’®. 


. Transitivity of induction: Let H and K be subgroups of G with H C K. If @ is a character of H, 


then (y*)° = @%. 


. Degree of induced character: If H is a subgroup of G and ¢ is a character of H, then the de- 


gree of the induced character g° equals the product of the index of H in G and the degree of 
p:%(e) =[G: H]g(e). 


. Let x be a character of G and let H be a subgroup of G. If the restriction xy is irreducible, 


then so is x. The converse of this statement does not hold. In fact, if H is the trivial subgroup 
then xq = x(e)1x, so any nonlinear irreducible character (e.g., x2 in Example 2 of section 68.4) 
provides a counterexample. 


. Let H bea subgroup of G and let y be a character of H. If the induced character g° is irreducible, 


then so is g. The converse of this statement does not hold (see Example 3). 


. Let H be a subgroup of G. If g is an irreducible character of H, then there exists an irreducible 


character x of G such that ¢ is a constituent of x7. 


. Frobenius Reciprocity: If x is a character of G and ¢ is a character of a subgroup H of G, then 


(v%, vale = (g, XH) H- 


. If x isa character of G and g isa character of a subgroup H of G, then (gxy)° = g° x. 
. Mackey’s Subgroup Theorem: If H and K are subgroups of G and ¢ is a character of H, then 


(@)r = Sees where T is a set of representatives for the (H, K)-double cosets in G 
(so that G = ewe, a disjoint union). 


. Ifg isa character of a normal subgroup N of G, then for eachs € G, the conjugate y* is a character 


of N. Moreover, y°(n) = v(sns~') (n € N). 

Clifford’s Theorem: Let N be a normal subgroup of G, let x be an irreducible character of G, and 
let g be an irreducible constituent of yy. Then yn = m war Qi, where ¢),..., @p are the distinct 
conjugates of g under the action of G and m = (xn, 9)n. 
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Examples: 


1. Given a subgroup H of G, the induced character (1;)° equals the permutation character corre- 
sponding to the action of G on the set of left cosets of H in G given by s(tH) = (st)H (s,t € G). 

2. The induced character (1,,))© equals the permutation character corresponding to the action of G 
on itself given by left multiplication. It is the character of the (left) regular representation of G. 
This character satisfies 


IG| ifs =e 


G = 
Me) = . ifs #e. 


3. Asan illustration of Frobenius Reciprocity (Fact 11), we have ((1te})%; Xe = Aeeys Xtep)tey = X Ce) 
for any irreducible character x of G. Hence, (hig)? = yenie x(e)x (cf. Fact 8 of section 68.4), 
that is, in the character of the regular representation (see Example 2), each irreducible character 
appears as a constituent with multiplicity equal to its degree. 


68.6 Representations of the Symmetric Group 


Definitions: 
Given a natural number n, a tuple a = [a,...,a)] of nonnegative integers is a (proper) partition of n 


(written a F n) provided 


* a; > a4; foralll <i <h, 
h 
© ea aj =n. 
The conjugate partition of a partition w | n is the partition a’ + n with ith component a; equal to 


the number of indices j for which a; > i. This partition is also called the partition associated with a. 
Given two partitions a = [a),...,a@,] and 6 = [6),..., 8] of n, a majorizes (or dominates) 6 if 


j j 
) aj > y B; 
=] =i 


for each 1 < j </h. This is expressed by writing a > 6 (or B X @). 

The Young subgroup of the symmetric group S,, corresponding to a partition a = [a),...,a,] of n 
is the internal direct product Sy = Sa, x --: x S,4,, where S4, is the subgroup of S,, consisting of those 
permutations that fix every integer not in the set 


aa {isksn 


i-1 i 
) aj< k < ; Oj 
j=l j=l 


(an empty sum being interpreted as zero). 
The alternating character of the symmetric group S,, is the character €,, given by 


(co) { 1 ifo is even, 
En(0) = 
—1. ifo is odd. 


Let G bea subgroup of S,, and let x bea character of G. The generalized matrix function d, :C"*" > C 
is defined by 


n 


d,(A) = S~ x(s) | Pais. 


sEG j=l 


When G = S,, and x is irreducible, d, is called an immanant. 
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Facts: 


The following facts can be found in [JK81, pp. 15, 35-37] or [Mer97, pp. 99-103, 214]. 


1. 
2s 


If x is a character of the symmetric group S,, then x (o) is an integer for eacho € S,,. 

Irreducible character associated with a partition: Given a partition @ of n, there is a unique irreducible 
character xq that is a constituent of both the induced character (1s, )S and the induced character 
((En)s,, )S«, The map a +> xq defines a bijection from the set of partitions of n to the set Irr(S,) 
of irreducible characters of S,. 

If w and £ are partitions of n, then the irreducible character x, is a constituent of the induced 
character (1s, )S» if and only if @ majorizes f. 

If a is a partition of n, then xy = €nXa- 

Schur’s inequality: Let x be an irreducible character of a subgroup G of S,. For any positive 
semidefinite matrix A € C"*", d,(A)/x(e) => det A. 


Examples: 
1. a = [5, 3%, 2, 13] (meaning [5, 3, 3, 2, 1, 1, 1]) is a partition of 16. Its conjugate is w’ = [7, 4, 3, 1, 1]. 
2. X{n) = 1s, and xj") = €n. 
3. In the notation of Example 3 of section 68.4, we have xo = X[4)) X1 = X14» X2 = X22) X3 = XA] 


and A= X([2,12]- 
According to Fact 4, a partition a of n is self-conjugate (meaning a’ = a) if and only if x,(0) = 0 
for every odd permutation o € S,,. 


. As an illustration of Fact 3, we have 


S 
(1s...) “= x) + XBa) + XB + X21 + X22 


The irreducible constituents of the induced character (1 ae )+ are the terms on the right-hand side 
of the equation. Note that [4], [3, 1], [27], and [2, 17] are precisely the partitions of 4 that majorize 
[2, 17] in accordance with the fact. 

When G = S,, and x = €, (the alternating character), d, (A) is the determinant of A € C”*”. 
When G = S, and x = lg (the principal character), d,(A) = >, 2¢ em ajs(;) is called the 
permanent of A € C”*", denoted per A. 

The following open problem is known as the Permanental Dominance (or Permanent-on-Top) 
Conjecture: Let x be an irreducible character of a subgroup G of S,,. For any positive semidefinite 
matrix A € C"*”, per A > d,(A)/x(e) (cf. Fact 5 and Examples 6 and 7). 
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One of the earliest surveys on nonassociative algebras is the article by Shirshov [Shi58] that introduced the 
phrase “rings that are nearly associative.” The first book in the English language devoted to a systematic study 
of nonassociative algebras is Schafer [Sch66]. A comprehensive exposition of the work of the Russian school 
is Zhevlakov, Slinko, Shestakov, and Shirshov [ZSS82]. A collection of open research problems in algebra, 
including many problems on nonassociative algebra, is the Dniester Notebook [FKS93]; the survey article 
by Kuzmin and Shestakov [KS95] is from the same period. Three books on Jordan algebras that contain 
substantial material on general nonassociative algebras are Braun and Koecher [BK66], Jacobson [Jac68], 
and McCrimmon [McC04]. Recent research appears in the Proceedings of the International Conferences on 
Nonassociative Algebra and Its Applications [Gon94], [CGG00], [SSS06]. The present chapter provides very 
limited information on Lie algebras, since they are the subject of Chapter 70. The last section (Section 69.9) 
presents three applications of computational linear algebra to the study of polynomial identities for 
nonassociative algebras: Pseudorandom vectors in a nonassociative algebra, the expansion matrix for a 
nonassociative operation, and the representation theory of the symmetric group. 


69.1 Introduction 


Definitions: 


An algebra is a vector space A over a field F together with a bilinear multiplication (x, y) xy from 
A x Ato A; that is, distributivity holds for alla,b € F andallx,y,z ¢€ A: 


(ax + by)z = a(xz) + b(yz), x(ay + bz) = a(xy) + b(xz). 


The dimension of an algebra A is its dimension as a vector space. 
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An algebra A is finite dimensional if A is a finite dimensional vector space. 
The structure constants of a finite dimensional algebra A over F with basis {x),...,x,} are the scalars 
ck € F (i, j,k =1,...,m) defined by: 


n 


= . k 
XjXj = Cj jXk- 


k=1 


An algebra A is unital if there exists an element 1 € A for which 
Ix =xl=x forallx eA. 
An involution of the algebra A is a linear mapping j: A > A satisfying 
j(Uj(x))=x and j(xy) = j(y)j(x) for allx,y € A. 


An algebra A is a division algebra if for every x, y € A with x 4 0 the equations xv = y and wx = y 
are solvable in A. 
The associator in an algebra is the trilinear function 


(x,y,z) = (xy)z — x(yz). 
An algebra A is associative if the associator vanishes identically: 
(x,y,z) =0 forallx,y,zeA. 


An algebra is nonassociative if the above identity is not necessarily satisfied. 
An algebra A is alternative if it satisfies the right and left alternative identities 


(y,x,x)=0 and (x,x,y)=0 for allx,y € A. 
An algebra A is anticommutative if it satisfies the identity 
x°=0 forallx€ A. 


(This implies that xy = — yx, and the converse holds in characteristic # 2.) 
The Jacobian in an anticommutative algebra is defined by 


J (x, y5Z) = (xy)z + (yz)x + (zx)y. 
A Lie algebra is an anticommutative algebra satisfying the Jacobi identity 
J(x,y,z)=0 forallx,y,zeA. 
A Malcev algebra is an anticommutative algebra satisfying the identity 
J (x,y,xz) = J(x,y,z)x forallx,y,ze€ A. 
The commutator in an algebra A is the bilinear function 
[x,y] = xy — yx. 


The minus algebra A~ of an algebra A is the algebra with the same underlying vector space as A but 
with [x, y] as the multiplication. 
An algebra A is commutative if it satisfies the identity 


xy =yx forallx,yeA. 
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A Jordan algebra is a commutative algebra satisfying the Jordan identity 
(x, yx) =0 forallx,yeA. 
The Jordan product (or anticommutator) in an algebra A is the bilinear function 
x* y= xXy+ YX. 


(The notation x o y is also common.) 

The plus algebra At of an algebra A over a field F of characteristic 4 2 is the algebra with the same 
underlying vector space as A but with x- y = $(x * y) as the multiplication. 

A Jordan algebra is called special if it is isomorphic to a subalgebra of At for some associative algebra 
A; otherwise it is called exceptional. 

Given two algebras A and B over a field F,a homomorphism from A to B isalinear mapping f: A > B 
that satisfies f(xy) = f(x) f(y) forallx,y € A. 

An isomorphism is a homomorphism that is a linear isomorphism of vector spaces. 

Let A be an algebra. Given two subsets B, C C Awe write BC for the subspace spanned by the products 
yz where y € B,z EC. 

A subalgebra of A is a subspace B satisfying BB C B. 

The subalgebra generated by a set S C A is the smallest subalgebra of A containing S. 

A (two-sided) ideal of an algebra A is a subalgebra B satisfying AB + BA C B. 

Given two algebras A and B over the field F, the (external) direct sum of A and B is the vector space 
direct sum A @ B with the multiplication 


(w,x)(y,z) =(wy,xz) forallw,ye¢ A andallx,ze B. 


Given an algebra A with two ideals B and C, we say that A is the (internal) direct sum of B and C if 
A= B @C (direct sum of subspaces). 


Facts: ([Shi58], [Sch66], [ZSS82], [KS95]) 


1. Every finite dimensional associative algebra over a field F is isomorphic to a subalgebra of a matrix 
algebra F"*” for some n. 
2. The algebra A~ is always anticommutative. If A is associative, then A™ is a Lie algebra. 
3. (Poincaré—Birkhoff—Witt Theorem or PBW Theorem) Every Lie algebra is isomorphic to a sub- 
algebra of A~ for some associative algebra A. 
4. Thealgebra A* is always commutative. If A is associative, then At isa Jordan algebra. (See Example 
2 in Section 69.9.) If A is alternative, then A* is a Jordan algebra. 
5. The analogue of the PBW theorem for Jordan algebras is false: Not every Jordan algebra is special. 
(See Example 4 below.) 
6. Every associative algebra is alternative. 
7. (Artin’s Theorem) An algebra is alternative if and only if every subalgebra generated by two elements 
is associative. 
8. Every Lie algebra is a Malcev algebra. 
9. Every Malcev algebra generated by two elements is a Lie algebra. 
10. If A is an alternative algebra, then A~ is a Malcev algebra. (See Example 3 in Section 69.9.) 
11. In an external direct sum of algebras, the summands are ideals. 


Examples: 


1. Associativity is satisfied when the elements of the algebra are mappings of a set into itself with the 
composition of mappings taken as multiplication. Such is the multiplication in the algebra End V, 
the algebra of linear operators on the vector space V. Every associative algebra is isomorphic to a 
subalgebra of the algebra End V, for some V. Thus, the condition of associativity of multiplication 
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characterizes the algebras of linear operators. (Note that End V is also denoted L(V, V) elsewhere 
in this book, but End V is the standard notation in the study of algebras.) 

Cayley—Dickson doubling process. Let A be a unital algebra over F with an involution x +> X 
satisfying 


x+x,xxeF forallxe A. (69.1) 


Leta € F,a #0. The algebra (A, a) is defined as follows: The underlying vector space is A® A, 
addition and scalar multiplication are defined by the vector space formulas 


(x1,%2) + (Wis y2) = (41 + yi, X2 +2), (x1, x2) = (€X1, cx2) for allc € F, (69.2) 


and multiplication is defined by the formula 


(x15 %2) (Mi 2) = (xin + ayr%, Miy2 + "1x2 ). (69.3) 
This algebra has an involution defined by 
(x1, X2) = (X], —x2). (69.4) 


In particular, starting with a field F of characteristic 4 2, we obtain the following examples: 


(a) The algebra C(a) = (F,a) is commutative and associative. If the polynomial x? + a is 
irreducible over F, then C(a) is a field, otherwise C(a) = F @ F (algebra direct sum). 


(b) The algebra H(a,b) = (C(a),b) is an algebra of generalized quaternions, which is asso- 
ciative but not commutative. 


(c) The algebra O(a, b,c) = (H(a, b),c) is an algebra of generalized octonions or a Cayley— 
Dickson algebra, which is alternative but not associative. (See Example 1 in Section 69.9.) 


The algebras of generalized quaternions and octonions may also be defined over a field of 
characteristic 2 (see [Sch66], [ZSS82]). 
Real division algebras [EHH91, Part B]. In the previous example, taking F to be the field R of real 
numbers anda = b = c = —1, we obtain the field C of complex numbers, the associative division 
algebra H of quaternions, and the alternative division algebra O of octonions (also known as the 
Cayley numbers). Real division algebras exist only in dimensions 1, 2, 4, and 8, but there are many 
other examples: The algebras C, H, and O with the multiplication x - y = xy are still division 
algebras, but they are not alternative and they are not unital. 
The Albert algebra. Let © be the octonions and let M3(Q) be the algebra of 3 x 3 matrices over 
O with involution induced by the involution of O, that is (a;;) > (jj). The subalgebra H3(Q) 
of Hermitian matrices in M3(O)* is an exceptional Jordan algebra, the Albert algebra: There is no 
associative algebra A such that H;(Q) is isomorphic to a subalgebra of A*. 


69.2 General Properties 


Definitions: 


Given an algebra A and an ideal I, the quotient algebra A/I is the quotient space A/J with multiplication 
defined by (x + I)(y+1I)=xy+I forallx,y € A. 

The algebra A is simple if AA 4 {0} and A has no ideals apart from {0} and A. 

The algebra A is semisimple if it is the direct sum of simple algebras. (The definition of semisimple 
that is used in the theory of Lie algebras is different; see Chapter 70.) 
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Set A! = A“ = A, and then by induction define 


Att — a A‘AD and AMM) = AMA — forn>1. 
i+j=nt+l 


The algebra A is nilpotent if A" = {0} for some n and solvable if A“) = {0} for some s. The smallest 
natural number n (respectively s) with this property is the nilpotency index (respectively, solvability 
index) of A. 

An element x € A is nilpotent if the subalgebra it generates is nilpotent. 

Anilalgebra (respectively nil ideal) is an algebra (respectively ideal) in which every element is nilpotent. 

An algebra is power associative if every element generates an associative subalgebra. 

An idempotent is an element e 4 0 of an algebra A satisfying e? = e. Two idempotents e, f are 
orthogonal if ef = fe = 0. 

For an algebra A over a field F, the degree of A is defined to be the maximal number of mutually 
orthogonal idempotents in the scalar extension F @; A, where F is the algebraic closure of F. 

The associator ideal D(A) of the algebra A is the ideal generated by all the associators. The associative 
center or nucleus N(A) of A is defined by 


N(A) = {x € A| (x, A, A) = (A, x, A) = (A, A, x) = {0} }. 
The center Z(A) of the algebra A is defined by 
Z(A) = {x € N(A)|[x, A] = {0}}. 
The right and left multiplication operators by an element x € A are defined by 
Ry: y > yx, Lyi y be xy. 


The multiplication algebra of the algebra A is the subalgebra M(A) of the associative algebra End A 
(of endomorphisms of the vector space A) generated by all R, and L, for x € A. 

The right multiplication algebra of the algebra A is the subalgebra R(A) of End A generated by all R, 
forxe€ A. 

The centroid C(A) of the algebra A is the centralizer of the multiplication algebra M(A) in the algebra 
End A; that is, 


C(A)={T € End A| TR, = RyT = Rey, Tly=LyT =Lry, — foranyx € A}. 


An algebra A over a field F is central if C(A) = F. 

The unital hull A? of an algebra A over a field F is defined as follows: If A is unital, then A? = A; and 
when A has no unit, we set A* = A @ F (vector space direct sum) and define multiplication by assuming 
that A is a subalgebra and the unit of F is the unit of A’. 

Let M be a class of algebras closed under homomorphic images. A subclass R of M is said to be 
radical if 


1. R is closed under homomorphic images. 
2. For each A € M there is an ideal R(A) of A such that R(A) € R and R(A) contains every ideal 
of A contained in R. 


3. R(A/R(A)) = {0}. 


In this case, we call the ideal R(A) the R-radical of A. The algebra A is said to be 7-semisimple if 
R(A) = {0}. 

If the subclass Nil of nil-algebras is radical in the class M, then the corresponding ideal Nil A, for 
A € M, is called the nil radical of A. In this case, the algebra A is called nil-semisimple if Nil A = {0}. By 
definition, Nil A contains all two-sided nil-ideals of A, and the quotient algebra A/Nil A is nil-semisimple, 
that is, Nil (A/Nil A) = {0}. 
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If the subclass Nilp of nilpotent algebras (or the subclass Solv of solvable algebras) is radical in the 
class M, then the corresponding ideal Nilp A (respectively Solv A), for A € M, is called the nilpotent 
radical (respectively the solvable radical) of A. 

For an algebra A over F, an A-bimodule is a vector space M over F with bilinear mappings 


AxM-—M, (x,m)t— xm and Mx A- M, (mx) mx. 


The split null extension E (A, M) of A by M is the algebra over F with underlying vector space A® M 
and multiplication 


(x+m)(y+n)=xy+(xn+my) forallx,y € A,mneM. 


For an algebra A, the regular bimodule Reg(A) is the underlying vector space of A considered as an 
A-bimodule, interpreting mx and xm as multiplication in A. 
If M is an A-bimodule, then the mappings 


p(x):mt> mx, Mx):m hr xm, 
are linear operators on M, and the mappings 
xr p(x), xb A(x), 


are linear mappings from A to the algebra Endy M. The pair (A, p) is called the birepresentation of A 
associated with the bimodule M. 

The notions of sub-bimodule, homomorphism of bimodules, irreducible bimodule, and faithful 
birepresentation are defined in the natural way. The sub-bimodules of a regular A-bimodule are exactly 
the two-sided ideals of A. 


Facts: ([Sch66], [Jac68], [ZSS82]) 


1. If Aisa simple algebra, then AA = A. 
2. (Isomorphism theorems) 


(a) If f: A— B isa homomorphism of algebras over the field F, then A/ker(F) = im(f) C B. 
(b) If B, and Bz are ideals of the algebra A with By C Bj, then (A/B2)/(B,/B2) = A/B). 


(c) If S is a subalgebra of A and B is an ideal of A, then BM S is an ideal of S and (B + S)/ 
~$/(BNS). 


3. The algebra A is nilpotent of index n if and only ifany product of n elements (with any arrangement 
of parentheses) equals zero, and if there exists a nonzero product of n — 1 elements. 

4. Every nilpotent algebra is solvable; the converse is not generally true. (See Example 1 below.) 

5. In any algebra A, the sum of two solvable ideals is again a solvable ideal. If A is finite-dimensional, 
then A contains a unique maximal solvable ideal Solv A, and the quotient algebra A/Solv A does 
not contain nonzero solvable ideals. In other words, the subclass Solv of solvable algebras is radical 
in the class of all finite dimensional algebras. 

6. An algebra A is associative if and only if D(A) = {0}, if and only if N(A) = A. 

Every solvable associative algebra is nilpotent. 

8. The subclass Nilp of nilpotent algebras is radical in the class of all finite dimensional associative 
algebras. 

9. A finite dimensional associative algebra A is semisimple if and only if Nilp A = {0}. 


ol 


10. The previous two facts imply that every finite dimensional associative algebra A contains a unique 
maximal nilpotent ideal N such that the quotient algebra A/N is isomorphic to a direct sum of 
simple algebras. 
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Over an algebraically closed field F , every finite dimensional simple associative algebra is isomorphic 
to the algebra F”*” of n x n matrices over F, for some n > 1. 


12. The subclass Nilp is not radical in the class of finite dimensional Lie algebras. (See Example 1 
below.) 

13. Over a field of characteristic zero, an algebra is power associative if and only if 

x?x=xx? and (x?x)x = x*x? for all x. 

14. Every power associative algebra A contains a unique maximal nil ideal Nil A, and the quotient 
algebra A/Nil A is nil-semisimple, that is, it does not contain nonzero nil ideals. In other words, 
the subclass Nil of nil-algebras is radical in the class of all power associative algebras. 

15. Fora finite dimensional alternative or Jordan algebra A we have Nil A = Solv A. 

16. For finite dimensional commutative power associative algebras, the question of the equality of the 
nil and solvable radicals is still open, and is known as Albert’s problem. An equivalent question is: 
Are there any simple finite dimensional commutative power associative nil algebras? 

17. Every nil-semisimple finite dimensional commutative power associative algebra over a field of 
characteristic # 2,3,5 has a unit element and decomposes into a direct sum of simple algebras. 
Every such simple algebra is either a Jordan algebra or a certain algebra of degree 2 over a field of 
positive characteristic. 

18. Direct expansion shows that these two identities are valid in every algebra: 

x(y,Z,w) + (x, y,z)w = (xy,z,w) — (x, yz,w) + (x, y,2w), 
[xy,z] — xly,z] —[x,z]ly = (x,y,z) — (% 2 y) + (4%, y). 
From these it follows that the associative center and the center are subalgebras, and 
D(A) = (A, A, A) + (A, A, AJA = (A, A, A) + AA, A, A). 
19. Ifz € Z(A), then for any x € A we have 
R,Ry = Ry Rz = Rix = Ruz. 

20. If A is unital, then its centroid C(A) is isomorphic to its center Z(A). If A is simple, then C(A) is 
a field which contains the base field F. 

21. Let A bea finite dimensional algebra with multiplication algebra M(A). Then 
(a) Ais nilpotent if and only if M(A) is nilpotent. 

(b) If A is semisimple, then so is M(A). 
(c) If A is simple, then so is M(A), and M(A) = End ci) A. 

22. An algebra A is simple if and only if the bimodule Reg(A) is irreducible. 

23. If Ais an alternative algebra (respectively a Jordan algebra), then its unital hull A? is also alternative 
(respectively Jordan). 

Examples: 

1. Let A be algebra with basis x, y, and multiplication given by x? = y? = 0 and xy = —yx = y. 
Then A is a Lie algebra and A?) = {0} but A” # {0} for any n > 1. Thus, A is solvable but not 
nilpotent. 

2. Let A be an algebra over a field F with basis x), x2, y, z and the following nonzero products of basis 


elements: 


YX) =AX|V=X, 2X. =AaXZ= xX, 
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where 0 4 a € F. Then I) = Fx; + Fx) + Fy and I, = Fx, + Fx) + Fz are different maximal 
nilpotent ideals in A. By choosing a = 1 or a = —1 we obtain a commutative or anticommutative 
algebra A. 

3. In general, in a nonassociative algebra, a power of an element is not uniquely determined. In the 
previous example, for the element w = x, + x. + y + z we have 


ww? =0 but w(ww*) = (1+a)(x,; + x). 
4, Let Aj,..., A, be simple algebras over a field F with bases 
{vi lie hy}, ..., WP |i © In}. 
Consider the algebra A = Fe @ A; @--- ® A, (vector space direct sum) with multiplication 
defined by the following conditions: 
(a) The A; are subalgebras of A. 
(c) ev! = vie = e for alli, j. 
(d) e =e. 


Then I = Fe is the unique minimal ideal in A, and I? = I. In particular, Solv A = {0}, but A is 
not semisimple (compare with the Lie algebra case). 

5. Suttles’ example. (Notices AMS 19 (1972) A-566) Let A be a commutative algebra over a field F 
of characteristic # 2, with basis x; (1 <i < 5) and the following multiplication table (all other 
products are zero): 


x1 X2 X2X4 x1 x5 X3,  X1X3 = X4y  XQ2XZ = Xs. 


Then A isa solvable power associative nil algebra that is not nilpotent. Moreover, Nil A = Solv A = 
A, and Nilp A does not exist (if F is infinite then A has infinitely many maximal nilpotent ideals). 


69.3 Composition Algebras 


Definitions: 


A composition algebra is an algebra A with unit 1 over a field F of characteristic 4 2 together with a 
norm n(x) (a nondegenerate quadratic form on the vector space A) that admits composition in the sense 
that 


n(xy) = n(x)n(y)  forallx € A. 


A quadratic algebra A over a field F is a unital algebra in which every x € A satisfies the condition 
x? € Span(x, 1). In other words, every subalgebra of A generated by a single element has dimension < 2. 
A composition algebra A is split if it contains zero-divisors, that is, ifxy = 0 for some nonzero x, y € A. 


Facts: ([Sch66], [Jac68], [ZSS82], [Bae02]) 


1. Every composition algebra A is alternative and quadratic. Moreover, every element x € A satisfies 
the equation 


x? — t(x)x + n(x) =0, 


where ¢(x) isa linear form on A (the trace) and n(x) is the original quadratic form on A (the norm). 
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2. For a composition algebra A the following conditions are equivalent: 
(a) A is split; 
(b) n(x) = 0 for some nonzero x € A; 
(c) A contains an idempotent e # 1. 


3. Let A be a unital algebra over a field F with an involution x +> * satisfying Equation 69.1 
from Example 2 of Section 69.1. The Cayley—Dickson doubling process gives the algebra (A, a) 
defined by Equations 69.2 to 69.4. It is clear that A is isomorphically embedded into (A,a) and 
that dim(A,a) = 2dim A. For v = (0,1), we have v? = a and (A,a) = A® Av. For any 
y=y+ypv € (A,a), we have ¥ = yj — yov. 

4. In a composition algebra A, the mapping x +> X = f(x) — x is an involution of A fixing the 
elements of the field F = F 1. Conversely, if A is an alternative algebra with unit 1 and involution 
x +> X satisfying Equation 69.1 from Example 2 of Section 69.1, then xX € F and the quadratic 
form n(x) = xX satisfies n(xy) = n(x)n(y). 

5. The mapping y +> j is an involution of (A,a) extending the involution x +» xX of A. Moreover, 
y + y and yy are in F for every y € (A,a). If the quadratic form n(x) = xX is nondegenerate 
on A, then the quadratic form n(y) = y y is nondegenerate on (A, a), and the form n(y) admits 
composition on (A, a) if and only if A is associative. 

6. Every composition algebra over a field F of characteristic # 2 is isomorphic to F or to one of the 
algebras of types 2a—2c obtained from F by the Cayley—Dickson process as in Example 2 of Section 
69.1. 

7. Every split composition algebra over a field F is isomorphic to one of the algebras F @ F, M)(F), 
Zorn(F ) described in Examples 2 to 4 below. 

8. Every finite dimensional composition algebra without zero divisors is a division algebra, and so 
every composition algebra is either split or a division algebra. 

9. Every composition algebra of dimension > 1 over an algebraically closed field is split, and so every 
composition algebra over an algebraically closed field F is isomorphic to one of the algebras F, 
F @ F, M,(F), Zorn(F ). 


Examples: 


1. The fields of real numbers R and complex numbers C, the quaternions H, and the octonions O, 
are real composition algebras with the Euclidean norm n(x) = xx. The first three are associative; 
the algebra O provides us with the first and most important example of a nonassociative alternative 
algebra. 

2. Let F bea field and let A= F © F be the direct sum of two copies of the field with the exchange 
involution (a,b) = (b,a), the trace t((a, b)) = a + b, and the norm n((a,b)) = ab. Then Aisa 
two-dimensional split composition algebra. 

3. Let A= M)(F) be the algebra of 2 x 2 matrices over F with the symplectic involution 


x= K> X= ; 
c d —c a 
the matrix trace t(x) = a+d,and the determinant norm n(x) = ad—bc.Then Aisa4-dimensional 
split composition algebra. 
4. An eight-dimensional split composition algebra is the Zorn vector-matrix algebra (or the Cayley— 


Dickson matrix algebra), obtained by taking A = Zorn(F), which consists of all 2 x 2 block 
matrices with scalars on the diagonal and 3 x 1 column vectors off the diagonal 


Zorn(F) = {s = i 


a,be F, uve Pt, 


69-10 Handbook of Linear Algebra 


with norm, involution, and product 


n(x) = ab — (u,v), t= | g = 
—V a 
a) a2 + (Uy, V2) a Uz + uy bz — vy X V2 
X1xX2 = 5 
Via + byv2 + uy X Uy by bz + (vj), uw) 


the scalar and vector products are defined for u = [u), u2, us|? andv = [v1,v2,v3]" by 


(u, Vv) = uv) + U2V2 + U3V3, u X v= [u2V3 — 43V2, U3V] — U1V3, Uy V2 — U2V1 J. 


Applications: 
1. If we write the equation n(x)n(y) = n(xy) in terms of the coefficients of the algebra elements 
x, y with respect to an orthogonal basis for each of the composition algebras R, C, H, O given in 
Example 3 of Section 69.1, then we obtain an identity expressing the multiplicativity of a quadratic 
form: 


(xp tes toxg) pte ty) Hate te 
Here the z; are bilinear functions in the x; and y;: To be precise, z; is the coefficient of the ith basis 


vector in the product of the elements x = (x),...,x,) and y = (y1,..., ¥~). By Hurwitz’ theorem, 
such a k-square identity exists only for k = 1, 2,4, 8. 


69.4 Alternative Algebras 


Definitions: 


A left alternative algebra is one satisfying the identity (x, x, y) = 0. 
A right alternative algebra is one satisfying the identity (y,x,x) = 0. 
A flexible algebra is one satisfying the identity (x, y,x) = 0. 
An alternative algebra is one satisfying all three identities (any two imply the third). 
The Moufang identities play an important role in the theory of alternative algebras: 


(xy -z)y= x(yzy) right Moufang identity 
(yzy)x = y(z- yx) left Moufang identity 
(xy)(zx) = x(yz)x central Moufang identity. 


(The terms yzy and x(yz)x are well-defined by the flexible identity.) 

An alternative bimodule over an alternative algebra A is an A-bimodule M for which the split null 
extension E (A, M) is alternative. 

Let A be an alternative algebra, let M be an alternative A-bimodule, and let (A, ¢) be the associated 
birepresentation of A. The algebra A acts nilpotently on M if the subalgebra of End M, which is generated 
by the elements A(x), p(x) for all x € A, is nilpotent. If y € A, then y acts nilpotently on M if the 
elements A(y), e(y) generate a nilpotent subalgebra of End M. 

A finite dimensional alternative algebra A over a field F is separable if the algebra Ax = K @ p Ais 
nil-semisimple for any extension K of the field F. 
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Facts: ([Sch66], [ZSS82]) Additional facts about alternative algebras are given in Section 69.1 and facts 
about right alternative algebras are given in Section 69.6: 


1. 
2. 


10. 


11. 


12. 


13. 


14. 


Every commutative or anticommutative algebra is flexible. 
Substituting x + z for x in the left alternative identity, and using distributivity, we obtain 


(x,z,y) + (z,x,y) =0. 


This is the linearization on x of the left alternative identity. Linearizing the right alternative identity 
in the same way, we get 


(y,x,z)+(y,z,x) =0. 


From the last two identities, it follows that in any alternative algebra A the associator (x, y, z) is a 
skew-symmetric (alternating) function of the arguments x, y, z. 


. Every alternative algebra is power associative (Corollary of Artin’s Theorem, Fact 7, Section 69.1). 


In particular, the nil radical Nil A exists in the class of alternative algebras. 
Every alternative algebra satisfies the three Moufang identities and the identities 


(x,V, ¥Z) = (x,y,z), (x,y, ZY) = y(x; y,Z). 


A bimodule M over an alternative algebra A is alternative if and only if the following relations hold 
in the split null extension E (A, M): 


(x,m,x)=0 and (x,m,y) = (m,y,x) =(y,x,m) forallx,ye¢ A and meM. 
It follows from the definition of alternative bimodule and the Moufang identities that 
[o(x),A(x)] =0, p(x") =(p(x))* fork>1, — A(x*) = (A(x))* fork > 1. 


This implies that any nilpotent element of an alternative algebra acts nilpotently on any bimodule. 
If every element of an alternative algebra A acts nilpotently on a finite dimensional alternative 
A-bimodule M, then A acts nilpotently on M. 

A nilpotent algebra A acts nilpotently on the A-bimodule M if and only if the algebra E(A, M) is 
nilpotent. 

Ina finite dimensional alternative algebra A, every nil subalgebra is nilpotent. In particular, the nil 
radical Nil A is nilpotent. 

The subclass Nilp of nilpotent algebras is radical in the class of all finite dimensional alternative 
algebras. For any finite dimensional alternative algebra A, we have 


Nil A = Solv A = Nilp A. 


Let A be a finite dimensional alternative algebra. The quotient algebra A/Nil A is semisimple, that 
is, it decomposes into a direct sum of simple algebras. Every finite dimensional nil-semisimple 
alternative algebra is isomorphic to a direct sum of simple algebras, where every simple algebra is 
either a matrix algebra over a skew-field or a Cayley—Dickson algebra over its center. 

Let A be a finite dimensional alternative algebra over a field F. If the quotient algebra A/Nil A is 
separable over F, then there exists a subalgebra B of A such that B is isomorphic to A/Nil A and 
A= B @Nil A (vector space direct sum). 

Every alternative bimodule over a separable alternative algebra is completely reducible (as in the 
case of associative algebras). 

Let A bea finite dimensional alternative algebra, let M be a faithful irreducible A-bimodule, and let 
(A, 0) be the associated birepresentation of A. Either M is an associative bimodule over A (which 
must then be associative), or one of the following holds: 
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(a) The algebra A is an algebra of generalized quaternions, A is a (right) associative irreducible 
representation of A, and p(x) = A(x) for every x € A. 


(b) The algebra A = O is a Cayley—Dickson algebra and M is isomorphic to Reg(O). 


15. Every simple alternative algebra (of any dimension) is either associative or is isomorphic to a 
Cayley—Dickson algebra over its center. 


69.5 Jordan Algebras 


In this section, we assume that the base field F has characteristic £ 2. 


Definitions: 


A Jordan algebra is a commutative nonassociative algebra satisfying the Jordan identity 
(x? y)x = x? (yx). 
The linearization on x of the Jordan identity is 
2((xz)y)x + (x*y)z = 2(xz)(xy) + x°(yz). 
A Jordan algebra J is special if it is isomorphic to a subalgebra of the algebra At for some associative 


algebra A; otherwise, it is exceptional. 


Facts: ([BK66], [Jac68], [ZSS82], [McC04]) Additional facts about Jordan algebras are given in 


Section 69.1, and facts about noncommutative Jordan algebras are given in Section 69.6: 


1. (Zelmanov’s Simple Theorem) Every simple Jordan algebra (of any dimension) is isomorphic to 
one of the following: (a) an algebra of a bilinear form, (b) an algebra of Hermitian type, (c) an 
Albert algebra. For definitions see Examples 3, 4, and 5 below. 

2. Let J be a Jordan algebra. Consider the regular birepresentation x > L,,x t+ R, of the algebra 
J . Commutativity and the Jordan identity imply that for all x, y € J we have 


Le=Re [Ro Rel=0, Rey — Ry Re +2RyRyRe —2ReRyx = 0. 


Linearizing the last equation on x we see that for all x, y,z € J we have 


Rixely — RyRuz + RyRy Re + ReRyRe — ReRyz — ReRyx = 0. 


3. For every k > 1, the operator R,« belongs to the subalgebra A C End J generated by R, and 
Ry. Since A is commutative, we have [R,:, R,«] = 0 for all k,@ > 1, which can be written as 
(hex = (0), 

4. It follows from the previous fact that every Jordan algebra is power associative and the radical Nil J 
is defined. 

5. Let J bea finite dimensional Jordan algebra. As for alternative algebras, we have 


NilJ = SolvJ = Nilp J, 


that is, the radical Nil J is nilpotent. The quotient algebra J /Nil J is semisimple, that is, isomorphic 
to a direct sum of simple algebras. If the quotient algebra J /Nil J is separable over F, then there 
exists a subalgebra B of J such that B is isomorphic to J /NilJ and J = B @ NilJ (vector space 
direct sum). 
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6. Ifa Jordan algebra J contains an idempotent e, the operator R, satisfies the equation R,(2R, — 1) 
(R, — 1) = 0, and the algebra J has the following analogue of the Pierce decomposition from the 
theory of associative algebras: 


J=J:@Jijy2@Jo, where Jj = Ji(e) = {x € J | xe = ix}. 
For i, j = 0,1 (i F j), we have the inclusions 
WSIn The Shp, FJ;=0}), He Shth. 


More generally, if J has unit 1 = 5~;' 


;—1 ¢i Where e; are orthogonal idempotents, then 


J= QD jij, where Jj; = Ji (ei), Jig =IiplenAIiplej;) fori F j, 


i<j 


and the components J;; are multiplied according to the rules 


Ji SSa Jyh SJ. J STut+Jij. Tadjq = {0} for distinct i, j; 


VigJ jx GJin  Jij Ske = {0}, JijJee = {0} for distinct i, j,k, £. 


7. Every Jordan algebra that contains >3 strongly connected orthogonal idempotents is special. (Or- 
thogonal idempotents e), e2 are strongly connected if there exists an element 412 € Jy. for which 
ur, =e, +e.) 

8. (Coordinatization Theorem) Let J be a Jordan algebra with unit 1 = ee e; (n > 3), where the 
e; are mutually strongly connected orthogonal idempotents. Then J is isomorphic to the Jordan 
algebra H,,(.D) of Hermitian n x n matrices over an alternative algebra D (which is associative for 
n > 3) with involution *« such that H(D,*) C N(D), where N(D) is the associative center of D. 

9. Every Jordan bimodule over a separable Jordan algebra is completely reducible, and the structure 
of irreducible bimodules is known. 


Examples: 


1. Thealgebra At. If A is an associative algebra, then the algebra At is a Jordan algebra. Every subspace 
J of A closed with respect to the operation x - y = d(xy + yx) isa subalgebra of the algebra AT 
and every special Jordan algebra J is (up to isomorphism) of this type. The subalgebra of A 
generated by J is called the associative enveloping algebra of J . Properties of the algebras A and 
A” are closely related: A is simple (respectively nilpotent) if and only if A* is simple (respectively 
nilpotent). 

2. The algebra At may bea Jordan algebra for nonassociative A; for instance, if A is a right alternative 
(in particular, alternative) algebra, then A* is a special Jordan algebra. 

3. Thealgebra ofa bilinear form. Let X be a vector space of dimension > 1 over F, with a symmetric 
nondegenerate bilinear form f(x, y). Consider the vector space direct sum J (X, f) = F @ X, and 
define on it a multiplication by assuming that the unit element 1 € F is the unit element of J (X, f) 
and by setting xy = f(x, y)1 for any x, y € X. Then J (X, f) isa simple special Jordan algebra; its 
associative enveloping algebra is the Clifford algebra C(X, f) of the bilinear form f. When F = R 
and f(x, y) is the ordinary dot product on X, the algebra J (X, f) is called a spin-factor. 

4. Algebras of Hermitian type. Let A be an associative algebra with involution +. The subspace 
H(A,*) = {x € A|x* = x} of *-symmetric elements is closed with respect to the Jordan 
multiplication x - y and, therefore, is a special Jordan algebra. For example, let D be an associative 
composition algebra with involution x +> X and let D’*” be the algebra of n x n matrices over D. 
Then the mapping S: (x;;) +> (Xj) is an involution of D’*” and the set of D-Hermitian matrices 
H,(D) = H(D"*", S) is a special Jordan algebra. If A is +-simple (if it contains no proper ideal 
I with I* C I), then H(A, *) is simple. In particular, all the algebras H,,(D) are simple. Every 
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algebra At is isomorphic to the algebra H(B,*«) where B = A @ A°PP (algebra direct sum) and 
(x1, x2)" = (x25%1). 

5. Albert algebras. If D = © is a Cayley—Dickson algebra, then the algebra H,,(O) of Hermitian 
matrices over D is a Jordan algebra only for n < 3. For n = 1,2 the algebras are isomorphic to 
algebras of bilinear forms and are, therefore, special. The algebra H3(O) is exceptional (not special). 
An algebra J is called an Albert algebra if K ®¢ J = H3(O) for some extension K of the field F. 
Every Albert algebra is simple, exceptional, and has dimension 27 over its center. 


69.6 Power Associative Algebras, Noncommutative Jordan 
Algebras, and Right Alternative Algebras 


A natural generalization of Jordan algebras is the class of algebras that satisfy the Jordan identity but which 
are not necessarily commutative. If the algebra has a unit element, then the Jordan identity easily implies 
the flexible identity. The right alternative algebras have been the most studied among the power associative 
algebras that do not satisfy the flexible identity. 

As in the previous section, we assume that F is a field of characteristic 4 2. 


Definitions: 


A noncommutative Jordan algebra is an algebra satisfying the flexible and Jordan identities. In this 
definition the Jordan identity may be replaced by any of the identities 


x? (xy) = x(x’y), (yx)x? = (yx*)x, (xy) x? = (x*y)x. 


A subspace V of an algebra A is right nilpotent if V’"’ = {0} for some n > 1, where V‘) = V and 
viet) = yiny, 


Facts: ([Sch66], [ZSS82], [KS95]) 


1. Let A be a finite dimensional power associative algebra with a bilinear symmetric form (x, y) 
satisfying the following conditions: 


(a) (xy,z) = (x, yz) forall x,y,z € A. 
(b) (e,e) # 0 for every idempotent e € A. 
(c) (x,y) = Oif the product xy is nilpotent. 


Then Nil A = Nil At = {x € A|(x,A) = {0}}, and if F has characteristic 4 2,3,5, then the 
quotient algebra A/Nil A is a noncommutative Jordan algebra. 

2. Let A bea finite dimensional nil-semisimple flexible power associative algebra over an infinite field 
of characteristic # 2,3. Then A has a unit element and is a direct sum of simple algebras, each 
of which is either a noncommutative Jordan algebra or (in the case of positive characteristic) an 
algebra of degree 2. 

3. The structure of arbitrary finite dimensional nil-semisimple power associative algebras is still 
unclear. In particular, it is not known whether they are semisimple. It is known that in this case 
new simple algebras arise even in characteristic zero. 

4. Analgebra A is a noncommutative Jordan algebra if and only if it is flexible and the corresponding 
plus-algebra A* is a Jordan algebra. 

5. Let A be a noncommutative Jordan algebra. For x € A, the operators R,, Lx, L.2 generate a 
commutative subalgebra in the multiplication algebra M(A), containing all the operators R,« and 
L,« fork > 1. 
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6. 
7 


Every noncommutative Jordan algebra is power associative. 

Let A be a finite dimensional nil-semisimple noncommutative Jordan algebra over F. Then A has 
a unit element and is a direct sum of simple algebras. If F has characteristic 0, then every simple 
summand is either a (commutative) Jordan algebra, a quasi-associative algebra (see Example 3 
below), or a quadratic flexible algebra. In the case of positive characteristic, there are more examples 
of simple noncommutative Jordan algebras. 


. Unlike alternative and Jordan algebras, an analogue of the Wedderburn Principal Theorem on 


splitting of the nil radical does not hold in general for noncommutative Jordan algebras. 
Every quasi-associative algebra (see Example 3 below) is a noncommutative Jordan algebra. 


10. Every flexible quadratic algebra is a noncommutative Jordan algebra. 
11. The right multiplication operators in every right alternative algebra A satisfy 
Re Re“ Rey = Re Ry, 
where x+y = t(xy + yx) is the multiplication in the algebra AT. (Recall that, in this section, we 
assume that the characteristic of the base field is 4 2.) 

12. If Aisaright alternative algebra (respectively, noncommutative Jordan algebra), then its unital hull 
A® is also right alternative (respectively, noncommutative Jordan). 

13. If Aisa right alternative algebra, then the mapping x +> Rx isa homomorphism of the algebra At 
into the special Jordan algebra R(A)*. If A has a unit element, then this mapping is injective. For 
every right alternative algebra A, the algebra At is embedded into the algebra R(A*)* and, hence, 
is a special Jordan algebra. 

14. Every right alternative algebra A satisfies the identity 

Rye Rye = Ryrre foranyxe A and k,€>1. 
Therefore, A is power associative and the nil radical Nil A is defined. 

15. Let A be an arbitrary right alternative algebra. Then the quotient algebra A/Nil A is alternative. In 
particular, every right alternative algebra without nilpotent elements is alternative. 

16. Every simple right alternative algebra that is not a nil algebra is alternative (and, hence, either 
associative or a Cayley—Dickson algebra). The nonnil restriction is essential: There exists a nonal- 
ternative simple right alternative nil algebra. 

17. A finite dimensional right alternative nil algebra is right nilpotent and, therefore, solvable, but such 
an algebra can be nonnilpotent. In particular, the subclass Nilp is not radical in the class of finite 
dimensional right alternative algebras. 

Examples: 
1. The Suttles algebra (Example 5 in Section 69.2) is a power associative algebra that is not a noncom- 
mutative Jordan algebra. For another example see Example 5 below. 
2. The class of noncommutative Jordan algebras contains, apart from Jordan algebras, all alternative 
algebras (and, thus, all associative algebras) and all anticommutative algebras. 
3. Quasi-associative algebras. Let A be an algebra over a field F and leta € F,a # 4. Define a new 


multiplication on A as follows: 
X-g y=axy+(1—a)yx, 


and denote the resulting algebra by A’. The passage from A to A® is reversible: A = (A*)? for 
b = a/(2a — 1). Properties of A and A® are closely related: The ideals (respectively subalgebras) 
of A are those of A‘; the algebra A® is nilpotent (respectively solvable, simple) if and only if the 
same holds for A. If A is associative, then A® is a noncommutative Jordan algebra; furthermore, 
if the identity [[x, y],z] = 0 does not hold in A, then A® is not associative. In particular, if A is 


69-16 Handbook of Linear Algebra 


a simple noncommutative associative algebra, then A‘ is an example of a simple nonassociative 
noncommutative Jordan algebra. The algebras of the form A* for an associative algebra A are 
split quasi-associative algebras. More generally, an algebra A is quasi-associative if it has a scalar 
extension, which is a split quasi-associative algebra. 

4. Generalized Cayley—Dickson algebras. For a),...,a, € F — {0} let 


A(a,)=(F,a,), ..., A(ay,...,4,) = (A(aq,...,4n—1), An), 


be the algebras obtained from F by successive application of the Cayley—Dickson process (Example 
2 of section 69.1). Then A(aj,...,a,) isa central simple quadratic noncommutative Jordan algebra 
of dimension 2”. 

5. Let V be a vector space of dimension 2n over a field F with a nondegenerate skew-symmetric 
bilinear form (x, y). On the vector space direct sum A = F @ V define a multiplication (as for 
the Jordan algebra of bilinear form) by letting the unit element 1 of F be the unit of A and by 
setting xy = (x, y)1 for any x, y € V. Then A isa simple quadratic algebra (and, hence, it is power 
associative), but A is not flexible and, thus, is not a noncommutative Jordan algebra. 


69.7 Malcev Algebras 


Some of the theory of Malcev algebras generalizes the theory of Lie algebras. For information about Lie 
algebras, the reader is advised to consult Chapter 70. 


Definitions: 


A Malcev algebra is an anticommutative algebra satisfying the identity 
T(x, y,xz) =J(x,y,z)x, where J (x,y,z) = (xy)z+(yz)x + (zx)y. 
In a left-normalized product we omit the parenthesis, for example, 
xYZX = ((xy)z)x, yzxx = ((yz)x)x. 


A representation of a Malcev algebra A is a linear mapping p: A —> End V satisfying the following 
identity for all x,y,z € A: 


plxy-z) = p(x)p(y)p(z) — plz)e(x)e(y) + ply) p(zx) — plyz) p(x). 


We call V a Malcev module for A. The anticommutativity of A implies that the notion of a Malcev module 
is equivalent to that of Malcev bimodule; we set xv = —vx forallx € A,v € V. 
The Killing form K (x, y) on a Malcev algebra A is defined (as for a Lie algebra) by 


K(x, y) = trace(R, Ry). 
Facts: ([KS95], [She00], [PS04], [SZ06]) 
1. After expanding the Jacobians, the Malcev identity takes the form 
XYZX + YZXX + ZXXY = XY+ XZ, 


(using our convention on left-normalized products). If F has characteristic # 2, the Malcev identity 
is equivalent to the more symmetric identity 


xyzt + yztx + ztxy + txyz=xz- yt. 


2. Any two elements in a Malcev algebra generate a Lie subalgebra. 
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10. 


11. 


12. 


The structure theory of finite dimensional Malcev algebras repeats the main features of the corre- 
sponding theory for Lie algebras. For any alternative algebra A, the minus algebra A~ is a Malcev 
algebra. Let O = O(a, b,c) be a Cayley—Dickson algebra over a field F of characteristic 4 2. Then 
O = F @ M (vector space direct sum), where M = {x € O|t(x) = 0}. The subspace M is a 
subalgebra (in fact, an ideal) of the Malcev algebra O-, and M = O”/F (in fact, O~ is the Malcev 
algebra direct sum of the ideals F and M). The Malcev algebra M = M(a, b,c) is central simple 
and has dimension 7 over F; if F has characteristic 4 3, then M is not a Lie algebra. 

Every central simple Malcev algebra of characteristic # 2 is either a Lie algebra or an algebra 
M(a, b,c). There are no non-Lie simple Malcev algebras in characteristic 3. 

Two Malcev algebras M(a,b,c), M(a’,b’,c’) are isomorphic if and only if the corresponding 
Cayley—Dickson algebras O(a, b,c), O(a’, b’, c’) are isomorphic. 

Let A be a finite dimensional Malcev algebra over a field F of characteristic 0 and let Solv A be 
the solvable radical of A. The algebra A is semisimple (it decomposes into a direct sum of simple 
algebras) if and only if Solv A = {0} (in fact, this is often used as the definition of “semisimple” for 
Malcev algebras, following the terminology for Lie algebras). If the quotient algebra A/Solv A is 
separable, then A contains a subalgebra B = A/Solv A and A = B @ Solv A (vector space direct 
sum). 

The Killing form K (x, y) is symmetric and associative: 


K(x,y) = K(y,x), K (xy, z) = K(x, yz). 


The algebra A is semisimple if and only if the form K(x, y) is nondegenerate. For the solvable 
radical we have Solv A = {x € A| K(x, A”) = {0}}. In particular, A is solvable if and only if 
K(A, A2) = {0}. 

If all the operators p(x) for x € Aare nilpotent, then they generate a nilpotent subalgebra in End V 
(A acts nilpotently on V). If the representation p is almost faithful (that is, ker p does not contain 
nonzero ideals of A), then A is nilpotent. 

Every representation of a semisimple Malcev algebra A is completely reducible. 

If A is a Malcev algebra and V is an A-bimodule, then V is a Malcev module for A if and only if 
the split null extension E (A, V) is a Malcev algebra. 

Let A be a Malcev algebra, and let V be a faithful irreducible A-module. Then the algebra A is 
simple, and either V is a Lie module over A (which must then be a Lie algebra) or one of the 
following holds: 


(a) A= M(a,b,c) and V isa regular A-module. 


(b) A is isomorphic to the Lie algebra s/(2, F) with dim V = 2 and p(x) = x*, where x* is 
the matrix adjoint to x € A C M)(F). (Here the matrix adjoint is defined by the equation 
x x* = x*x = det(x)J, where I is the identity matrix.) 


The speciality problem for Malcev algebras is still open: Is every Malcev algebra embeddable into the 
algebra A~ for some alternative algebra A? This is the generalization of the Poincaré—Birkhoff—Witt 
theorem for Malcev algebras. 


69.8 Akivis and Sabinin Algebras 


The theory of Akivis and Sabinin algebras generalizes the theory of Lie algebras and their universal 
enveloping algebras. For information about Lie algebras, the reader is advised to consult Chapter 70. 


Definitions: 


An Akivis algebra is a vector space A over a field F,, together with an anticommutative bilinear operation 
Ax A— A denoted [x, y], and a trilinear operation A x A x A — A denoted (x, y, z), satisfying the 
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Akivis identity for all x, y,z € A: 


[[x, y],z] + [ly.z], x] + Uz. x], y] = (x,y,z) + (25x) + (4.x, y) — (4529) — (52) — (zy x). 


A Sabinin algebra is a vector space A over a field F,, together with multilinear operations 


(X15: 8s XS VES) (m > 0), 
satisfying these identities: 
(Xios ti ioy) =O; (69.5) 
(X15. Xpy Uy Vy Xp py oy Xm3 Vs Z) — (Ky. Xp V5 Uy Xp py + Xm3 Vs Z) 
r 
+ s So ays wey Xs(k)s (Xs(kL)> ++ +9 Xs(r)3 Uy V9 Xptiy ++ +> Xm3 Y>Z) = 0, (69.6) 
k=0 5 


F 
Kuyyw ix, 6 Xp UZ V,W) + a Se Getiys £9 X5(k)3 (Xs(kt1)y ++ +9 Xs(r)3Vvw),U)] =O, (69.7) 
k=0 


Ss 


where s is a (k,r — k)-shuffle (a permutation of 1,...,7r satisfying s(1) <--- < s(k) ands(k+1) < 

- < s(r)) and the operator K,,,,, denotes the sum over all cyclic permutations. (See Fact 9 below for an 
alternative formulation of this definition.) 

An algebra Fy4[X] from a class M, with a set of generators X, is called the free algebra in M with 
the set X of free generators, if any mapping of X into an arbitrary algebra A € M extends uniquely to a 
homomorphism of Fy4[X] to A. 

Let I be any subset of Fy,[X]. The T-ideal in Fy,[X] determined by I, denoted by T = T(I, X), 
is the smallest ideal of Fy,[X] containing all elements of the form f(x,,...,x,) for all f € I and all 
X1y 20+ Xn € Fry [X]. 


Facts: ([HS90], [She99], [SU02], [GH03], [Per05], [BHP05], [BDE05]) 
1. Free algebras may be constructed as follows. Let S be a set of generating elements and let Q be a 
set of operation symbols. Let r:Q — N (the nonnegative integers) be the arity function, that is, 
w € Qwill represent an n-ary operation for n = r(w). The set W(S, Q) of nonassociative Q-words 
on the set S is defined inductively as follows: 


(a) SC W(S,Q). 
(b) Ifw € Qand x,...,x, € W(S, Q) where n = r(w), then w(x),...,xX,) € W(S,Q). 


Let F bea field and let F (S, &) be the vector space over F with basis W(S, Q). For each w € Q 
we define an n-ary operation with n = r(w) on F(S, Q), denoted by the same symbol a, as follows: 
Given any basis elements x1,..., x, € W(S, Q), we set the value of w on the arguments x),..., Xn 
equal to the nonassociative word w(x),...,X,), and extend linearly to all of F (S, Q). The algebra 
F(S, Q) is the free 82-algebra on the generating set S over the field F with the operations w € Q. 

2. The quotient algebra F(S,Q)/ T(J, S,Q) is the free M-algebra for the class M = M(I) of 
{2-algebras defined by the set of identities I. 

3. Every subalgebra of a free Akivis algebra is again free. 

4. Every Akivis algebra is isomorphic to a subalgebra of Akivis(A) for some nonassociative algebra 
A. This generalizes the Poincaré—Birkhoff—Witt theorem for Lie algebras. The free nonassociative 
algebra is the universal enveloping algebra of the free Akivis algebra. (See Example 1 below.) 

5. The free nonassociative algebra with generating set X has a natural structure of a (nonassociative) 
Hopf algebra, generalizing the Hopf algebra structure on the free associative algebra. The Akivis 
elements (the elements of the subalgebra generated by X using the commutator and associator) 
are properly contained in the primitive elements (the elements satisfying A(x) = x ®@1+1@x 
where A is the co-multiplication). The Akivis elements and the primitive elements have a natural 
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structure of an Akivis algebra. The primitive elements have the additional structure of a Sabinin 
algebra. 

6. The Witt dimension formula for free Lie algebras (the primitive elements in the free associative 
algebra) has a generalization to the primitive elements in the free nonassociative algebra. 

7. Sabinin algebras are a nonassociative generalization of Lie algebras in the following sense: The 
tangent space at the identity of any local analytic loop (without associativity assumptions) has a 
natural structure of a Sabinin algebra, and the classical correspondence between Lie groups and Lie 
algebras generalizes to this case. 

8. Every Sabinin algebra arises as the subalgebra of primitive elements in some nonassociative Hopf 
algebra. 

9. Another (equivalent) way to define Sabinin algebras, which exploits the Hopf algebra structure, is 
as follows. Let A be a vector space and let T(A) be the tensor algebra of A. We write A: T(A) > 
T(A) ® T(A) for the co-multiplication on T(A): the algebra homomorphism that extends the 
diagonal mapping A:ut 1@u+u@1 foru € A. We will use the Sweedler notation and write 
A(x) = 33 xq) @ xX) for any x € T(A). Then A isa Sabinin algebra if it is equipped with a trilinear 
mapping 


T(A)@A@QA-— A, XBYOzZY (x y,zZ), for x € T(A) and y,zé A, 


satisfying the identities 


(x;y, y) = 0, (69.8) 
(x @u@v@x'sy,z) — (x @v@u@x'sy,z) + So (xa) ® (x(2)3 U,V) @ x's y,Z) 

= 0, (69.9) 
Rysrin [« @usv,w) + Soa (xayivsw),t)] =0, (69.10) 


where x,x’ € T(A) and u,v,w,y,z € A. Identities (8 to 10) exploit the Sweedler notation to 
express identities (5 to 7) in a more compact form. 


Examples: 


1. Any nonassociative algebra A becomes an Akivis algebra Akivis( A) if we define [x, y] and (x, y,z) 
to be the commutator xy — yx and the associator (xy)z — x(yz). If A is an associative algebra, then 
the trilinear operation of Akivis(A) is identically zero; in this case the Akivis identity reduces to the 
Jacobi identity, and so Akivis( A) is a Lie algebra. If A is an alternative algebra, then the alternating 
property of the associator shows that the right side of the Akivis identity reduces to 6(x, y, z). 

2. Every Lie algebra is an Akivis algebra with the identically zero trilinear operation. Every Malcev 
algebra (over a field of characteristic 4 2, 3) is an Akivis algebra with the trilinear operation equal 
to 2 J (x, 52). 

3. Every Akivis algebra A is a Sabinin algebra if we define 


(a,b) = —[a, bl, (x3a,b) = (x,b,a) — (x, a,b), (X1,...5Xn3a,b) = 0 (m > 1), 
for all a,b,x,x; € A. 
4. Let L bea Lie algebra with a subalgebra H C L andasubspace V C L for which L = H @ V. We 


write Py: L — V for the projection onto V with respect to this decomposition of L. We define an 
operation 


(-,3-):TV)@V@V>V 
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by (using the Sweedler notation again) 


{x @a@b}+ S- { x(1) ® (x(2)34, b) } = 0, 
where for x = x; ®--- @ x, € T(V) we write 


{x} = Py([x1[.--> [%n—-1%n]] +++]. 


Then the vector space V together with the operation (—,—;—) is a Sabinin algebra, and every 
Sabinin algebra can be obtained in this way. 


69.9 Computational Methods 


For homogeneous multilinear polynomial identities of degree n, the number of associative monomials is 
n\ and the number of association types is C,, (the Catalan number); hence, the number of nonassociative 
monomials grows superexponentially: 


1 ie = ‘ CAS oes 
nl-— = —— sn". 
n\n—-1 (n — 1)! 

One way to reduce the size of the computations is to apply the theory of superalgebras [Vau98]. 
Another technique is to decompose the space of multilinear identities into irreducible representations 
of the symmetric group S,,. The application of the representation theory of the symmetric group to the 
theory of polynomial identities for algebras was initiated in 1950 by Malcev and Specht. The computational 
implementation of these techniques was pioneered by Hentzel in the 1970s [Hen77]; for detailed discussions 
of recent applications see [HP97] and [BH04]. Another approach has been implemented in the Albert 


system [Jac03]. In this section, we present three small examples (n < 4) of computational techniques in 
nonassociative algebra. 


Examples: 


1. The identities of degree 3 satisfied by the division algebra of real octonions. 
There are 12 distinct multilinear monomials of degree 3 for a nonassociative algebra: 


(xy)z, (xz)y, (yx)z, (yz)x, (zx) y, (zy)x, x(yz), x(zy), y(xz), y(zx), z(xy), z(yx). 


We create a matrix of size 8 x 12 and initialize it to zero; the columns correspond to the nonasso- 
ciative monomials. We use a pseudorandom number generator to produce three octonions x, y, z 
represented as vectors with respect to the standard basis 1,1, j,k, 2, m, n, p. We store the evaluation 
of monomial j in column j of the matrix. For example, generating random integers from the set 
{—1,0, 1} using the base 3 expansion of 1//2 gives 


x= [1,—1,0,-1,—-1,1,0,0], y = [-1,1,1,1L0,0,1,0], z = [-1,1,1,1,0,0,0,—1]. 


Evaluation of the monomials gives the matrix in Table 69.1; its reduced row echelon form appears 
in Table 69.2. The nullspace contains the identities satisfied by the octonion algebra: the span of 
the rows of the matrix in Table 69.3. These rows represent the linearizations of the right alternative 
identity (row 1), the left alternative identity (row 2), and the flexible identity (row 5), together with 
the assocyclic identities (x, y,z) = (y,z,x) and (x, y,z) = (z,x, y) (rows 3 and 4). 

2. The identities of degree 4 satisfied by the Jordan product x * y = xy + yx in every associative algebra 
over a field of characteristic 0. 
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TABLE 69.1 The octonion evaluation matrix 


3 9 9 3 3 9 3 9 9 3 3 9 

1 5) 3 ) 1 1 3 1 1 1 —5 3 
-l 1 =) > 1 3 1 1 1 3 3 bi] 
2 2 -4 —2 0 —2 2 2 -4 -2 0 —2 

1 1 1 Oo. 3 > 3 3 7 5 1 

3 3 3 1 bs) 1 3 3 3 1 é] 1 
—-10 —2 0 4 2 4 -8 -4 —2 6 4 2 
0 0 0 6.22) 2-2 2 2 2 4 —4 0 


TABLE 69.2 The reduced row echelon form of the octonion 
evaluation matrix 


100 0 0 0 0 1 Tf Shed 1 
0 1°00 0 0 0 1 0 0 0 0 
0 0 10 0 0 0 0 1 0 0 0 
0 0 0 10 0 0 0 0 1 0 0 
0 0 0 0 1 0 0 0 0 0 1 0 
0 00 0 0 1 0 0 0 0 0 1 
0 000001 -1 -!1 1 al 


TABLE 69.3 A basis for the nullspace of the octonion 
evaluation matrix 


-1 -l 0 0 0 0 1 10 0 0 0 
-l 0 -il 0 0 0 1010 0 0 
1 0 0 -il 0 0 -1 001 0 0 
1 0 0 0 -il 0 -1 0001 «0 
-l 0 0 0 Qo-=1 100001 


The operation x xy satisfies commutativity in degree 2, and there are no new identities of degree 3, so 
we consider degree 4. There are 15 distinct multilinear monomials for a commutative nonassociative 
operation, 12 for association type ((——)—)— and 3 for association type (——)(—-—): 


((wxx)*xy)*Z, ((Wxx)KzZ)KY, ((WxY)*x)KZ, ((WKY)KZ)RX, ((WHZ) KX) KY, 
((wxz)Ky)*x, ((xxy)aw)*zZ, ((xxy)*z)aw, ((x*z)KW)KY, ((XKZ) KY) RW, 
( 


((yx*z)ax)xw, ((yxz)ewW)&x, (Wx )&(y*Z), (WxY)*(~RZ), (WZ)K(XKY). 


When each of these monomials is expanded in terms of the associative product, there are 24 possible 
terms, namely the permutations of w, x, y, zin lexicographical order: wxyz,...,zyxw. We construct 
a 24 x 15 matrix in which the i, j entry is the coefficient of the ith associative monomial in the 
expansion of the jth commutative monomial (see Table 69.4). The nontrivial identities of degree 
4 satisfied by x * y correspond to the nonzero vectors in the nullspace. The reduced row echelon 
form appears in Table 69.5. The rank is 11 and, so, the nullspace has dimension 4. A basis for the 
nullspace consists of the rows of Table 69.6. The first row represents the linearization of the Jordan 
identity; this is the only identity that involves monomials of both association types. (This proves 
that the plus algebra A* of any associative algebra A is a Jordan algebra.) The Jordan identity 
implies the identities in the other three rows, which are permuted forms of the identity 


wx«(xx(yz)) — x*(wx(yz)) = (we(xexy))*zZ — (x(Wxy))&Z + ye (We (X*Z)) — yx(x*«(WZ))5 


that is, the commutator of multiplication operators is a derivation. 
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TABLE 69.4 The Jordan expansion matrix in degree 4 


100000110001 i210 «0 
010000001282 0 1 10 0 
001000121201 00 0 i210 
0 0 0 000001 101 0 
0 0 0 0 00111000021 
0000010 21 0 «0 100 1 
101 0 00 0 0 0 0 1 0 0 
0 1 0 0 1 0 0 0 0 0 1 0 0 
0 0 1 011000000021 
0000010 21 0 «0 100 1 
0 0 0 10010000 1 0 
0 0 0 000001 101 0 
1110000010 00%0i210 
0 0 0 100 10000 1 0 
1 10 0 01000000 0i21 
0 0 0 0 00111000021 
0 1 0 0 100 0010 41 0 0 
010000000121 01 1 0 0 
1 10 0 01000000021 
00110 11000000 0 «21 
1 110000010 00021 0 
001000121201 00 0 i210 
101 100000010210 0 
10000041100 0i1i1 0 «0 


TABLE 69.5 The reduced row echelon form of the Jordan 
expansion matrix 


100000000 — 0 0 0 0 0 

00000 0 0 1 0 0 1 
001000 0 0 0 0 0 -1 0 0 0 
00010 0 0 0 0 1 0 0 1 
0000100 0 0 0 0 -1 0 0 0 
000001000 — 0 0 0 0 0 
000000 i210 0 0 - 0 0 0 0 
000000 0i21 «(0 1 0 0 1 
0000000 0 1 0 - 0 0 0 0 
000000 0 0 0 0 0 0 10 -1 
000000 0 0 0 0 0 0 0 1 -1 


TABLE 69.6 A basis for the nullspace of the Jordan expansion 


matrix 
0 -1 0 -1 00 0 -1 000012141 
0 -1 0 -1 00 41 -1 10 10 0 0 0 
0 -1 1 -1 10 0 -1 00 0410 0 0 
1 -1 0 -1 01 0 -1 0100 0 0 0 


3. The identities of degree 4 satisfied by the commutator [x,y] = xy — yx in every alternative algebra 
over a field of characteristic zero. 

The group algebra QS,, decomposes as a direct sum of full matrix algebras of size d, x d, where 
the index A runs over all partitions 4 of the integer n; here d, is the dimension of the irreducible 
representation of S,, corresponding to the partition A. We choose the “natural representation” to fix 
a particular decomposition. For each A there is a projection p, from QS, onto the matrix algebra of 
size d, x d,. In the case n = 4 the partitions and the dimensions of the corresponding irreducible 
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TABLE 69.7 Partitions of 4 and irreducible 

representations of S4 

“jy. ££ 31 om 2 dill 
dy, 1 3 2 3 1 


representation S, are given in Table 69.7. For a nonassociative operation in degree 4 there are 5 
association types: 


(ese Sen Seas Sa a Sea), 


and so any nonassociative identity can be represented as an element of the direct sum of 5 copies 
of QS,,: Given a partition A, the nonassociative identity projects via p, to a matrix of size d, x 5d). 
For an anticommutative operation in degree 4 there are 2 association types: 


Nis=taLbal Teall 


and so any anticommutative identity projects via p, to a matrix of size d, x 2d,. The linearizations 
of the left and right alternative identities are 


L(x, y,Z) = (xy)z — x(yz) + (yx)z — y(xz); R(x, y,z) = (xy)z — x(yz) + (xz)y — x(zy). 
Each of these can be “lifted” to degree 4 in five ways; for L (a, b,c) we have 
wL(x,y,z), L(xw,y,z), L(x,yw,z), L(x,y,zw), L(x, y,z)w; 


and similarly for R(x, y, z). Altogether we have 10 lifted alternative identities that project via p, to 
a matrix of size 10d, x 5d,. Using the commutator to expand the two anticommutative association 
types gives 


[[[x, y],z],w] = ((xy)z)w — ((yx)z)w — (2(xy))w + (z(yx))w 
—w((xy)z) + w((yx)z) + w(2(xy)) — w(z(yx)), 


[[x, y], [z, w]] = (xy) (zw) — (yx) (zw) — (xy)(wz) + (yx) (wz) 
—(zw)(xy) + (zw) (yx) + (wz) (xy) — (wz)(yx). 


Given a partition A we can store these two relations in a matrix of size 2d, x 7d,: We use all 
7 association types, store the right sides of the relations in the first 5 types, and —I (J is the 
identity matrix) in type 6 (respectively 7) for the first (respectively second) expansion. For each 
partition A, all of this data can be stored in a matrix A, of size 12d, x 7d, which is schematically 
displayed in Table 69.8. We compute the reduced row echelon form of A,: Let 7 be the largest 
number for which rows 1 — i of RREF(A;) have a nonzero entry in the first 5 association types. 
Then the remaining rows of RREF(A,) have only zero entries in the first 5 types; if one of these 


TABLE 69.8 The matrix of Malcev identities for 
partition 2 


0 0 

Lifted alternative identities : 

Ay = 0 0 
Expansion of [[[x, y],z],w] —I 0 


Expansion of [[x, y], [z, w]] Oost 
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TABLE 69.9 The lifted and expansion identities for partition A = 22 


—2 


0 


TABLE 69.10 The reduced row echelon form of the lifted 


and expansion identities 


0 0 0 0 


0 0 0 0 


0 0 0 0 


0 0 0 0 


0 0 0 0 


0 0 0 0 


0 0 0 


0 
-l1 


-1 


0 00 0 0 0 0 


000000 0 
=, 


1 


0 0 0 


0 
-l 


000 0 0 
1 


0 
0 0 0 0 


0000 0 
1 al 


0 0 0 0 0 


0 


0 0 0 


0 


0 


0 00 000 0 0 


0 00 00 0 0 0 


0 00000 0 0 


0 00 00 0 0 0 


TABLE 69.11 The anticommutative identities for partition 


dA = 22 


2 
-l 
0 
0 
0 
0 


0000000 0 0 «0 
0000000 0 0 0 
0000000 0 0 0 
0000000 0 0 0 
0000000 0 0 0 
0000000 0 0 ~0 


TABLE 69.12 The reduced row echelon form of the 


anticommutative identities 


0 0 0 
0 ol 
0 0 0 


1 
0 


0000000 0 0 «0 
0000000 0 0 0 
0000000 0 0 ~0 


| 
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rows contains nonzero entries in the last 2 types, this row represents an identity that is satisfied by 
the commutator in every alternative algebra. However, such an identity may be a consequence of 
the obvious anticommutative identities: 


[[{x, y],z],w] + [[ly,x],z],w] =0, 
[[x, y], [z,w]] + (ly, x], [z,w]] = 0, [[x, y], [z,w]] + [[z,w], [x, y]] = 0. 


These identities are represented by a matrix of size 3d, x 7d, in which the first 5d, columns are 
zero. We need to determine if any of the rows i + 1 to 12d, of RREF(A,) do not lie in the row 
space of the matrix of anticommutative identities. If such a row exists, it represents a nontrivial 
identity satisfied by the commutator in every alternative algebra. For example, consider the partition 
A = 22 (d, = 2). The 24 x 14 matrix A, for this partition appears in Table 69.9, and its reduced 
row echelon form appears in Table 69.10. The 6 x 14 matrix representing the anticommutative 
identities for this partition appears in Table 69.11, and its reduced row echelon form appears in 
Table 69.12. Comparing the last four rows of Table 69.10 with Table 69.12, we see that there is one 
new identity for 4 = 22 represented by the third-last row of Table 69.10. Similar computations 
for the other partitions show that there is one nontrivial identity for partition A = 211 and no 
nontrivial identities for the other partitions. The two identities from partitions 22 and 211 are the 
irreducible components of the Malcev identity: The submodule generated by the linearization of 
the Malcev identity (in the S4-module of all multilinear anticommutative polynomials of degree 4) 
is the direct sum of two irreducible submodules corresponding to these two partitions. 
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A Lie algebra is a (nonassociative) algebra satisfying x? = 0 for all elements x of the algebra (which 
implies anticommutativity) and the Jacobi identity. Lie algebras arise naturally as (vector) subspaces of 
associative algebras closed under the commutator operation [a,b] = ab — ba. The finite-dimensional 
simple Lie algebras over algebraically closed fields of characteristic zero occur in many applications. This 
chapter outlines the structure, classification, and representation theory of these algebras. We also give 
examples of other types of algebras, e.g., one class of infinite-dimensional simple algebras and one class 
of finite-dimensional simple Lie algebras over fields of prime characteristic. Section 70.1 is devoted to 
general definitions about Lie algebras. Section 70.2 discusses semisimple and simple algebras. This section 
includes the classification of finite-dimensional simple Lie algebras over algebraically closed fields of char- 
acteristic zero. Section 70.3 discusses module theory and includes the classification of finite-dimensional 
irreducible modules for the aforementioned algebras as well as the explicit construction of some of these 
modules. Section 70.4 discusses graded algebras and modules and uses this formalism to present results 
on dimensions of irreducible modules. 


70.1 Basic Concepts 


Unless specified otherwise, F denotes an arbitrary field. All vector spaces and algebras are over F. The 
reader is referred to Chapter 69 for definitions of many basic algebra terms. 


Definitions: 


Let A be an algebra over a field F. An automorphism is an algebra isomorphism of A to itself. The set of 
all automorphisms of A is denoted Aut(A). 

A linear transformation D: A — A isa derivation of A if D(ab) = D(a)b + aD(b) for alla,b € A. 
The set of all derivations of A is denoted Der(A). 

Let A be an associative algebra and X be a subset of A. Then the smallest ideal of A containing X is 
denoted by < X > and called the ideal generated by X. 

Let V be a vector space over a field F. Let V®" denote the tensor product of n copies of V and set 
T(V) = Fl+ Ye V®" Define a linear map T(V) @ T(V) > T(V),u@ve w,bylu=ul=u 
for all u € T(V) and 


(V1 @... @ Vm) (Uy ®@ ... @ Un) = V1, @... QO Vn @ Uy @...@ Uy 
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whenever m,n > 1,V1,...5VinxsU1s---»Un € V. T(V) is the tensor algebra on the vector space V. This 
algebra is defined, and denoted (54) V, in Section 13.9. 

An algebra L over a field F with product [, ]: L x L > L, (a,b) + [a,b] isa Lie algebra if it satisfies 
both 


[a,a] =0 
and 
[a, [b,c]] + [b, [c,a]] + [c, [a,b]] = 0 (Jacobi identity) 
for all a,b,c € L. The first condition implies 
[a,b] = —[b, a] (anticommutativity) 


and is equivalent to anticommutativity if the characteristic of F 4 2. 

A Lie algebra L is abelian if [a,b] = 0 for alla,b € L. 

If A is an algebra, the vector space A together with the product [, ] : A x A — A defined by 
[a,b] = ab — ba isan algebra denoted by A-. 

Let L bea Lie algebra. Let I denote the ideal in T(L) (the tensor algebra on the vector space L ) generated 
by {a ®b —b @a— [a,b]|a,b € L}. The quotient algebra T(V)/I is called the universal enveloping 
algebra of L and is denoted by U(L). 

Let V be a vector space with basis X. Define a mapi : X > T(V)> byt: x exe VEC 
Fi+ a ye" = T(V)-. Let Fr(X) be the Lie subalgebra of T(V)~ generated by 1(X). Fr(X) is called 
the free Lie algebra generated by X. 

Let V be a vector space and let I be the ideal in T(V) generated by {a ® b — b @ala,b € V}. The 
quotient T(V)/I is called the symmetric algebra on V and denoted by S(V). The image of V®” in S(V) 
is denoted by S"(V). An equivalent construction of this algebra (as a subalgebra, denoted \/ V, of T(V)) 
is given in Section 13.9. 

Let V be a vector space and let I be the ideal in T(V) generated by {a © ala € V}. The quotient 
T(V)/I is called the exterior algebra on V and denoted by A(V). The image of a, ® ... ® aj is denoted 
by a, A... A a; and the image of V®" in A(V) is denoted by A"(V). An equivalent construction of this 
algebra (as a subalgebra, denoted /\ V, of T(V)) is given in Section 13.9. 

Let End V denote the vector space of all linear transformations from V to V (also denoted L(V, V) 
elsewhere in this book). Let L be a Lie algebra. If a € A, the map ad : L — End L defined by 
ad(a):b++ [a,b] for all b € L is called the adjoint map. 

Let F bea field of prime characteristic p and let L be a Lie algebra over F. If for every a € L there is 
some element a!?! € L such that (ad(a))? = ad(a!?!), then L is called a p-Lie algebra. 


Facts: 
The following facts (except those with a specific reference) can be found in [Jac62, Chap. 5]. 


1. [Jac62, p.6] If A isan associative algebra, then A~ isa Lie algebra. If F isa field of prime characteristic 
p> then A~ isa p-Lie algebra. 

2. Let A be an algebra over F. Then Der(A) is a Lie algebra. If F is a field of prime characteristic p, 
then Der(A) is a p-Lie algebra. 

3. Let V be a vector space. The tensor algebra T(V) has the structure of an associative algebra and 
T(V)~ isa Lie algebra. 

4. Let L bea Lie algebra. A subspace I C L is an ideal of L if [a,b] € I whenever a € A,b € I. The 
quotient space L/I with the product [a + I,b + I] = [a,b] + I isa Lie algebra. 

5. Universal property of U(L): Let L be a Lie algebra. Define a mapi: L > U(L)byt:arae 
L®! Cc FL+ yo L®" = T(L) > U(L). Then is a (Lie algebra) homomorphism of L into 
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U(L)~. If A is any associative algebra with unit 1 and ¢ is a homomorphism of L into A™, then 
there is a unique homomorphism w : U(L) > A such that w(1) = land¢@ = We. 

6. Poincaré—Birkhoff—Witt Theorem: Let L be a Lie algebra with ordered basis {l;|i € I}. Then 
{li,...li,|k => 0,1) <...< ig} isa basis for U(L). Consequently, 1: L + U(L) is injective. 

7. Universal property of the free Lie algebra: If L is any Lie algebra and if @ : X — L is any map, there 
is a unique homomorphism of Lie algebras y : Fr(X) — L such that @ = Wu. 

8. Universal property of S(V): Let V be a vector space. Define a mapi: V > S(V)byt:arhae 
V8lC FI+ Pa ve" _s S(V). Let A be an commutative associative algebra and ¢ : V > A 
bea linear map. Then there is a unique algebra homomorphism wy : S(V) — Asuch that @ = wu. 

9. Structure of S(V): Let V be a vector space with ordered basis B = {b;|i € I}. Then S(V) has basis 
{b;, ...b;,|1 > 0,1) < ... < i}. Consequently, S$(V) is isomorphic to the algebra of polynomials 
on B. 

10. [Lam01, p. 12] Structure of A(V): Let V bea vector space with ordered basis B = {b;|i € I}. Then 
A(V) has basis {b;, A...A0;,|1 > 0,1) < ... < i;}. Consequently, if V has finite dimension /, then 
dim A(V) = 2!, 


Examples: 


1. (End V)~ isa Lie algebra, denoted g/(V). Similarly, (F"*”)~ is a Lie algebra, denoted gl(n, F). 
These algebras are isomorphic. 

2. sl(n, F) = {x € gl(n, F)|tr(x) = 0} is a Lie subalgebra, and in fact, a Lie ideal of gl(n, F). 

3. Let L be a vector space with basis {e, f,h}. Defining [e, f] = h, [h,e] = 2e, and [h, f] = —2f 


gives L the structure of a Lie algebra. The linear map L — s1(2, F) defined bye t> : A »fR 


i 1 shire E | is an isomorphism of Lie algebras. 


70.2 Semisimple and Simple Algebras 


Definitions: 


Let L be a Lie algebra. The subspace spanned by all products [a,b],a,b € L, is a subalgebra of L. It is 
called the derived algebra of L and is denoted by L. 

Let L bea Lie algebra. For n > 2, L™ is defined to be (L“-)))™ and is called the nth-derived algebra 
of L. 

A Lie algebra L is solvable if L“” = {0} for some n > 1. 

Let L be a Lie algebra. The sum of all the solvable ideals of L is the radical of L and denoted Rad(L). 
(In Section 69.2, Rad(L) is called the solvable radical and denoted Solv L.) 

A Lie algebra L is semisimple if Rad(L) = {0}. N.B. This is standard terminology in the study of Lie 
algebras, but does not always coincide with the definition of semisimple given for nonassociative algebras 
in Section 69.2; cf. Fact 5 and Example 3. 

A Lie algebra L is simple if L contains no nonzero proper ideals and L" {0}. (The second condition 
excludes the one-dimensional algebra.) 

Let A, B be Lie algebras. Then the vector space A @ B can be given the structure of a Lie algebra, called 
the direct sum of A and B and also denoted A © B, by setting [a, + b, a2 + bz] = [a), a2] + [b1, be] for 
aj,a2 € A, by, by EB. 

Let L be a finite-dimensional Lie algebra. The Killing form, «,, is the symmetric bilinear form on L 
defined by x; (a, b) = tr((ad(a))(ad(b))). 

For V a finite dimensional vector space over an algebraically closed field F and x € End V, x is 
semisimple if the minimum polynomial of x has no repeated roots. 
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Let L be a Lie algebra over an algebraically closed field F and x € L. x is ad-nilpotent if ad(x) is a 
nilpotent linear transformation of L and x is ad-semisimple if ad (x) is a semisimple linear transformation 
of L. 

Let L be a Lie algebra. A subalgebra T C L is a torus if every element of T is ad-semisimple. 

Let T be a torus in L and let a € T*, the dual of T. Define L., the a-root space of L by 


Ly = {x € L|[t,x] =a(t)x Vt € T}. 


A vector space E over R with an inner product (i.e., a positive definite symmetric bilinear form)(., .) is 
a Euclidean space. 

Let E be a Euclidean space. 

ForOAx€ Eandye Eset<y,x >= wee. 

For 0 #4 x € E define o,, the reflection in the hyperplane orthogonal to x, by 


Ox(y) =y-< yx >x 


forall y € E. 
A finite subset R C E that spans E and does not contain 0 is a root system in E if the following three 
conditions are satisfied: 


¢ Ifx € R,a € R,andax € R,thena =+1. 
¢ Ifx € R,theno,R=R 
° Ifx,y € R, then < x,y >e Z. 


Let R be a root system in E. The rank of R is dim E. 

A root system R is decomposable if R = R, U R2 with @ # Ry, Ro C Rand (Rj, Ry) = {0}. If R is not 
decomposable, it is indecomposable. 

Let R bea root system in E. The subgroup of End E generated by {o,|a € R} is called the Weyl group 
of R. 

Let R be a root system in E. A subset B C R isa base for R if B is a basis for E and every x € R may 
be written 


x= So kyb 


beB 


where all k, > O or all k, < 0. 

Let R be a root system in E with base B and let w € R. @ is a positive root ifa = )7,., kpb where all 
k, = 0. Denote the set of positive roots by Rt. 

Let B = {b,,...,b;} bea base for a root system R. The matrix [< bj, b; >| , is called the Cartan matrix 
of R with respect to B. 


Facts: 
Most of the following facts (except those with a specific reference) can be found in [Hum72, pp. 35-65]. 


1. The radical, Rad(L), is a solvable ideal of L. 

2. For V a finite dimensional vector space over an algebraically closed field F,x € End V issemisimple 
if and only if x is similar to a diagonal matrix. 

3. [Jac62, p. 69] Cartan’s Criterion for Semisimplicity: Let L be a finite-dimensional Lie algebra over a 
field of characteristic 0. Then L is semisimple if and only if «;, is nondegenerate. 

4. [Jac62, p. 74] Let L bea finite-dimensional semisimple Lie algebra over a field of characteristic zero 
and let D € Der(L). Then D = ad(a) for somea € L. 

5. [Jac62, p. 71] Let L be a finite-dimensional semisimple Lie algebra over a field of characteristic 0. 
Then L is a direct sum of simple ideals. 
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6. 


7. 


10. 


11. 


12. 


Let L be a finite-dimensional Lie algebra over an algebraically closed field of characteristic 0. Any 
torus of L is abelian. 

Let L bea finite-dimensional semisimple Lie algebra over an algebraically closed field of character- 
istic 0. Let T be a maximal torus of dimension / in L and A = {a € T*|a 4 0, Ly  {0}}. Then: 


* dimL, =1forallac A. 
°*L= TO) en ba 


* Fora € A there exists f, € T such that a(t) = «1 (ty, t) for all t € T. Then, defining (a, 6) = 
K1 (ty, tg) gives the R span of A the structure of a Euclidean space of dimension / and A is a root 
system of rank / in this space. 


* A is indecomposable if and only if L is simple. 


¢ A spans T* and so each w € W acts on T*. 


. Let R bea root system in E with Weyl group W. Then W is finite and R has a base. Furthermore, if 


B,, By are two bases for R, then B} = w(B2) for some w € W. Consequently, when B, and By are 
appropriately ordered, the Cartan matrix of R with respect to B is the same as the Cartan matrix 
of R with respect to B2. Thus, we may refer to the Cartan matrix of R. 

Let L be a semisimple Lie algebra over a field of characteristic 0, let T,, I; be maximal tori in L, 
and let A;, A» be the corresponding root systems. Then there is an automorphism @ € Aut(L) 
such that @(T,) = Th. Consequently, when bases for A, and A) are appropriately ordered, the 
Cartan matrix for A, is the same as the Cartan matrix for A. Thus, we may refer to the Cartan 
matrix of L. 

Let L,, L2 be semisimple Lie algebras over an algebraically closed field of characteristic 0. If the 
Cartan matrices of L; and L, coincide, then L, and L» are isomorphic. 

Let M = [m;,;] bethe/ x! Cartan matrix of an indecomposable root system, with base appropriately 
ordered. Then the diagonal entries of M are all 2 and one of the following occurs. 


* Mis of type A; for some! > 1: m,,; = —lif|i— j) = 1m,; =Oif|i — j| > 1. 


* Misoftype B; forsome! > 3:my_1) = —23m;,; = —lifli—j| = land@, j) 4 U—-1,]);m,j; = 
Oif|i—j| > 1. 


* Misoftype C; forsome! > 2:mjj—) = —23mj,; = —lifli—j| = land(@, j) 4 (1-1); mij = 
Oif|i—j| > 1. 

* Mis of type D; for some! > 4: mj; = —1if |i — j] = land Gj) 4 7 — 1,1), (L1 — 1) or 
if @,j) = (7 — 2,1), 0,1 — 2);m,; = 0 if |i — j| > land G,j) ¢ ( — 2,1), (1 — 2) or if 

* Misoftype E),/ = 6,7,8:mj,; = —lifli—j| = landi, j 4 2orif(i, 7) = (1, 3)(3, 1)(2, 4), (4, 2); 
mj = Oif |i — j| > 1,4, 7) ¥ 0,3), D2, 4), (4, 2) or if |i — j| = Landi = 2 or j = 2. 


2 -1 0 0 


* Misoftype Fy: M = 


2 -il 
* Misoftype G2: M= ; 
ype G2 EB q 


Let L be a finite-dimensional simple Lie algebra over an algebraically closed field of characteristic 
zero. Then L is determined up to isomorphism by its root system with respect to any maximal 
torus. The Cartan matrix of the root system is of type A;,/ > 1; Bj,1 > 3;C),1 > 2;D),1 > 
4; E6, Ey, Es, F4, or G>. 
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Examples: 


1. 
2. 
3. 


eee , : 0 
. Let (.,.) be the skew-symmetric bilinear form on F 2! with matrix | 


. Let (.,.) be the symmetric bilinear form on F 2! with matrix |: 


The set R? with [u,v] = u x v (vector cross product) is a three dimensional simple Lie algebra. 
The set of all upper triangular complex n x n matrices is a solvable subalgebra of g/(n, C). 

[Pol69, p. 72] Let p > 3 be a prime. The only ideals of gl(p,Z,) are 0 C scal(p,Zp) C 
sl(p,Zp) C gl(p, Zp), where scal(p,Z,) is the set of scalar matrices. Thus, the only ideals of 
L = gl(p,Zp)/scal(p,Z,) are L, S = sl(p,Z,)/scal(p,Z,) and {0}. By considering D = 
diag(0,1,2,..., p — 1), we see that [s!(n, Zp), s1(n,Zp)] = sl(n, Zp), so [S, S] = S. Thus, S is 
not solvable and Rad(L) = 0. But L cannot be the sum of simple ideals. 


. The following are Cartan matrices of type A3, B3, C3 respectively: 


. Ifn > 1 and ifn is not a multiple of the characteristic of F, let L = sI(n, F). Then L is a simple 


Lie algebra of dimension n? — 1. If T denotes the set of diagonal matrices in s/(n, F) andife; € T* 
is defined by €;(diag(di,...,dn)) = dj, then T is a maximal torus in s/(n, F ), A, the set of roots 
of sl(n, F) with respect to T, is {e; — €j|i 4 j}, and the root space Lee, = FE;j,j.In addition, 
{€; — &41|1 <i <n -— 1} isa base for A and so the Cartan matrix of s/(n, F) is of type Ay_1. 

1 0 0O 


. Let (.,.) be the symmetric bilinear form on F2/+! with matrix |0 0 Jj|. Then L = 


0 oO 

{x € gl(2I+1, F)|(xu,v) = —(u,xv)Vu,v € F7+1) isa Liesubalgebra of g/(2/-+1, F ), denoted by 
o(21+1, F). Ifthe characteristic of F is not 2, it isa simple algebra of dimension 21 241. Let T denote 
the set of diagonal matrices in 0(2] + 1, F). Ife; € T* is defined by €;(diag(d},...,d141)) = di, 
and if vy; = €;4, for 1 <i <I, then T is a maximal torus in 0(2/ + 1, F); A, the set of roots 
of 0(2/ + 1, F) with respect to T, is {tvyj|l1 <i < J}U{tyjtvj|l <i 4 j < J}, and, for 
1<iAj <I,L, = F(Eyuin — Eig) Ly, = F(Eiini — Engitii)s Ly-v, = F( Bist jai — 
Eng jsiititi)s Loja, = F(Bisguiejgi — Ejtuititi) Lv, = FCEnisijo — Eitj+iiti)- In 
addition, {v; — vj44]1 < 1 < 1 — 1} U {1} isa base for A, and so the Cartan matrix of 0(2/] + 1, F) 
is of type By. 


I 
5 ps Then 1 = {xe 


gl(2l, F)|(xu,v) = —(u,xv)Vu,v € F7\ isa Lie subalgebra of gl (21, F ), denoted by sp(2I, F ). If 
the characteristic of F is not 2, it is a simple algebra of dimension 21 24]. Let T denote the set of 
diagonal matrices in sp(2I, F). Ife; € T* is defined by €;(diag(d),...,d1)) = dj, and if uw; = €; 
for 1 <i </, then T is a maximal torus in sp(2l, F), A, the set of roots of sp(21, F ) with respect to 
T,is{+2y;|1 <i <)}U{tpy; tp jli ¥ j}, and, for] <i <1, Lo, = FEj4i, L-oy, = F Elsi, 
Luj-w; = FCEi3 — Ersjasi)s Lun; = F(Eing + Ejsi), L-p-u; = F(Enij + Eiji). In 
addition, {u; — Wi41|1 <i <1] — 1} U {2,7} is a base for A, and so the Cartan matrix of sp(21, F) 
is of type C). 


jf Then = {x € gl(2l, F)| 
1 


(xu,v) = —(u,xv) Vuv € F”} is a Lie subalgebra of g/(21, F), denoted by o(2I, F). If the 
characteristic of F is not 2, it is a simple algebra of dimension 21? — I. Let T denote the set of 
diagonal matrices in 0(21, F). Ife; € T* is defined by €;(diag(d),...,d1)) = dj, and if vy; = €; 
for 1 <i <1, then T is a maximal torus in o0(2I, F), A, the set of roots of 0(21, F ) with respect to 
T, is {+v; SE vj|1 <i x j ics 1}, and, for 1 <i x f] <I, Ly,-v, = F(E;,; = E}4j1+i)» Ly 4y; = 
F(Ejj+j — EF jti), L—v,-v, = F (Ensi,j — E14j,). Inaddition, {vj —vj4i]1 <i < 1—-1U{y_1 +7} 
is a base for A, and so the Cartan matrix of 0(21, F ) is of type D;. 
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9. Let V bea vector space of dimension n > 1 over a field of characteristic 0. Let W(n) = Der(S(V)). 
Then W(n) is an infinite-dimensional simple Lie algebra. 

10. Let F be a field of characteristic p > 0. Let V be a vector space of dimension n > 1 with 
basis {x,,...,x,}. Let I denote the ideal < Re akoueye >C S(V), B(n : 1) denote S(V)/I, and 
W(n: 1) = Der(B(n: 1)). Then W(n : 1) is a p-Lie algebra of dimension np”. It is a simple Lie 
algebra unless p = 2 andn = 1. 


70.3. Modules 


Definitions: 


Let A be an associative algebra and V be a vector space over F. A representation of A on V is a homo- 
morphism ¢: A> End V. 

Let L bea Lie algebra and V bea vector space over F. A representation of L on V is a homomorphism 
o:L— gl(V). 

Let B be an associative algebra or a Lie algebra. A representation ¢ : B — gl(V) is reducible if there 
is some nonzero proper subspace W C V such that 6(x)(W) C W for all x € B. If ¢ is not reducible, it 
is irreducible. 

Let L be a Lie algebra and M be a vector space. M is an L-module if there is a linear map L ®@ M > 
M,a ® m+ amsuch that [a,b]m = a(bm) — b(am) foralla,b Ee L,me M. 

Let M be an L-module. A subspace N C M isa submodule of M if LN CN. 

Let M bean L-module. M is reducible if M contains a nonzero proper submodule. If M is not reducible 
it is irreducible. If M is a direct sum of irreducible submodules, it is completely reducible. 

Let M be an L-module and X be a subset of M. The submodule of M generated by X is the smallest 
submodule of M containing X. 

Let L bea Liealgebraand M, N be L-modules. A linear transformation ¢@ : M — Nisahomomorphism 
of L-modules if @(xm) = x@(m) for all x € L,m € M. The set of all L-module homomorphisms from 
M to N is denoted Hom(M, N). 

Let L be a finite-dimensional semisimple Lie algebra over an algebraically closed field of characteristic 
0. Let T be a maximal torus in L and M be an L-module. For A € T* define M,, the A-weight space of 
M, to be {m € M|tm = X(t)mVt € T}. 

Let L bea finite-dimensional simple Lie algebra over an algebraically closed field of characteristic 0. Let 
T be a maximal torus, A the corresponding root system, B a base for A, and At the corresponding set of 
positive roots. Let M be an L-module and A € T*. An element 0 4 m € M, isa highest weight vector of 
weight A if L,m = 0 for alla € AT. 


Facts: 


Unless specified otherwise, V denotes a vector space over a field F. 
The following facts may be found in [Hum, Sect. 6]. 


1. Let L be a Lie algebra and @ : L — gl(V) bea representation of L on V. Then V may be given 
the structure of an L-module by setting xv = $(x)(v) for all x € L,v € V. Conversely, if M is an 
L-module, then the map ¢@ : L — gl(M) defined by $(x)(m) = xmis a representation of L on 
M.A representation @ is irreducible if and only if the corresponding module is. 

2. Let ¢ bea representation of a Lie algebra L on V. Then, by the universal property of the universal 
enveloping algebra, @ extends to a representation of U(L) on V. Conversely, every representation of 
U(L) on V restricts to a representation of L on V. A representation ¢ of U(L) on V is irreducible 
if and only if its restriction to L is. 

3. Let L be a Lie algebra, M be an L-module, and N C M be a submodule. Then the quotient 
space M/N may be given the structure of an L-module by setting x(m + N) = xm-+ N for all 
xeLmeM. 
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. Let L bea Lie algebra and M, N be L-modules. Then the vector space M © N may be given the 


structure of an L-module by setting x(m +n) = xm-+<xn forallx eL,me M,neN. 


. Let L bea Lie algebra and M, N be L-modules. Then the vector space M @ N may be given the 


structure of an L-module by setting x(m @ n) = xm@n+m®@xn forallxe Lime MneN. 


. Let L bea Lie algebra and M, N be L-modules. Then Hom(M, N) may be given the structure of 


an L-module by setting (x@)(m) = x@(m) — $(xm) for allx e Lsme M. 


. Let L bea Lie algebra and V be an L-module. Then T(V) is an L-module and the ideals occurring 


in the definitions of S(V) and A(V) are submodules. Hence, $(V) and A(V) are L-modules. 
Furthermore, each S”(V) is a submodule of S(V) and each A"(V) is a submodule of A(V). 


. [Jac62, p. 79] Weyl’s Theorem: Let L be a finite-dimensional semisimple Lie algebra over a field of 


characteristic zero and let M be a finite-dimensional L-module. Then M is completely reducible. 


. [Hum72, pp. 107-114] Let L be a finite-dimensional semisimple Lie algebra over an algebraically 


closed field of characteristic 0 and M bea finite-dimensional L-module. Let T be a maximal torus in 
L, Abe the corresponding root system, B = {a,..., a} abase for A, and At be the corresponding 
set of positive roots. Then: 


* M= dere Mo. 


2 


(a; ,0%; ) for 


* M contains a highest weight vector of weight A for some A € T* and setting hj = 
1 <i <1, wehaveA(h;) > 0,A(h;) € Zfor1 <i </. 


° If M is irreducible and my, my are highest weight vectors corresponding to 41,42 € T*, then 
Ay =A, and Fm, = Fm). 


* If M, N are irreducible finite-dimensional L-modules containing highest weight vectors corre- 
sponding to the same A € T*, then M and N are isomorphic. 


° Let A € T™ satisfy A(h;) € Z,A(h;) > 0 for 1 <i <1. Then there exists a finite-dimensional 
L-module with highest weight A. 


Examples: 


Ti 


Let V be a vector space of dimension n > 1. Then V is a gl(V)-module and, hence, a gl(n, F )- 
module. Therefore, for each k > 0, S*(V) and A*(V) are modules for gl(V) and, thus, modules 
for any subalgebra of g/(n, F). 

Let V bea vector space with basis {x, y}. Let {e, f, h} bea basis for s1(2, F) with [e, f] = h, [h,e] = 
2e, [h, f] = —2f. The linear map s1(2, F) — Der(F[x, y]) defined by e ae i veoh i 
au - a is an isomorphism of s/(2, F) into Der(F [x, y]). Consequently, F [x,y] = S(V) is an 
s1(2, F)-module and each S"(V) is an s1(2, F) submodule. $"(V) has basis {x",x"~ly,..., y"} 
and so is an (n + 1)-dimensional s/(2, F )-module. It is irreducible. 


70.4 Graded Algebras and Modules 


Definitions: 


Let V be a vector space and A be an additive abelian group. For each a € A, let Vy be a subspace of V. If 
V = @aca Vo; then V is an A-graded vector space. 

Let B be an algebra and an A-graded vector space. B is an A-graded algebra if B, Bg © By+, for all 
a,Be A. 

Let B be an A-graded associative algebra or an A-graded Lie algebra. Let M be a B-module and an 
A-graded vector space. M is an A-graded module for B if By Mg © My+, for alla, B € A. 
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Let V be an A-graded vector space. V has graded dimension if dim(V,) < oo for alla € A. In this 
case, we define the graded dimension of V to be the formal sum 


gr dim(V) = )> dim(V,)t*. 


acA 


The graded dimension of V is sometimes called the character of V. 


Facts: 
1, 


[FLM88, Sect. 1.10] Let V, W be A-graded vector spaces with graded dimensions. Then V @ W isa 
graded vector space with graded dimension and gr dim(V ® W) = gr dim(V)+ gr dim(W). If W 
is a subspace of V and W, C V, foralla € A, then the quotient space V/ W has graded dimension 
and gr dim(V/W) = gr dim(V) — gr dim(W). If {(a, B) € A x AlVa, We # {0},a+ 8 = y} 
is finite for all y € A, then V @ W is a graded vector space with graded dimension, where 
(V@W), = irre Va ® Wg and gr dim(V ® W) = (gr dim(V))(gr dim(W)) (where 


we set t%t8 = ¢¢+8), 


. [Bou72, p. 36] Let V be a vector space with basis X. Setting Fr(X); = Fr(X)N ye gives 


Fr(X) the structure of a graded Lie algebra. If |X| = 1 is finite, Fr(X) has graded dimension and 
gr dim(Fr(X)) => eae e (d)17)t", where jz is the Mobius function (ie., (p)... Pr) = 


n>0n 


(—1)' if p,,..., p, are distinct primes and y(n) = 0 if p?|n for some prime p). 


. [Jac62, Sect. VIII.3] Let L be a finite-dimensional semisimple Lie algebra over an algebraically 


closed field of characteristic 0, T be a maximal torus in L, A be the corresponding root system, 
W be the Weyl group, At be the set of positive roots with respect to some base, and M be a 
finite-dimensional L-module. Then: 


* Let ZA denote the additive subgroup of T* generated by A. Then the root space decomposition 
L=T+) yea La gives L the structure of aZA-graded Lie algebra. Here Ly = T. 


* The weight space decomposition M = )* 
graded dimension. 


weT~ Ma gives M the structure ofa graded module with 


* Weyl character formula: Assume M is irreducible with highest weight A. Let 6 = +>, cq+ @. 
Then 


gr dim(M) = (Secor) / (Seow). 


wew wew 


Examples: 


i 


2: 


Setting T(V); = ye gives T(V) the structure of a Z-graded algebra. If V has finite dimension /, 
then T(V) has graded dimension and gr dim(T(V)) = (1—It)7!. 
Setting S(V); = Si(V) gives S(V) the structure of a Z-graded algebra. If V has finite dimension 
I, then S(V) has graded dimension and gr dim(S(V)) = (1 — ty. 


. Setting A(V); = Ai(V) gives A(V) the structure of a Z-graded algebra. If V has finite dimension 


I, then A(V) has graded dimension and gr dim(A(V)) = (1+ ty). 


. Let L bea finite-dimensional semisimple Lie algebra over an algebraically closed field of charac- 


teristic 0, T be a maximal torus in L, A be the corresponding root system, and B = {a@,..., a} 
be a base. For 1 <i < / define A; by hil tts) = 6;,;. Then: 
yy 


° If L = sl(V), where V is an/ + 1-dimensional vector space and the base for A is as described in 
Example 5 of section 70.2, then A‘(V) is the irreducible s1(V)-module of highest weight 4; for 
Pet. 
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° If L = o(V), where V is a 2/ + 1-dimensional vector space and the base for A is as described in 
Example 6 of section 70.2, then A'(V) is the irreducible 0(V)-module of highest weight A; for 
1<i<I-1. 

° If L = o(V), where V is a 2/-dimensional vector space and the base for A is as described in 


Example 8 of section 70.2, then A'(V) is the irreducible 0(V)-module of highest weight A; for 
1<i<1-—-2. 
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Matiap® is generally recognized as the leading software for scientific computation. It was originally devel- 
oped in the 1970s by Cleve Moler as an interactive Matrix Laboratory with matrix routines based on the algo- 
rithms in the LINPACK and EISPACK software libraries. In the original 1978 version everything in MATLAB 
was done with matrices, even the graphics. MATLAB has continued to grow and expand from the classic 1978 
Fortran version to the current version, MATLAB 7, which was released in May 2004. Each new release has 
included significant improvements. The graphics capabilities were greatly enhanced with the introduction 
of Handle Graphics and Graphical User Interfaces in version 4 (1992). A sparse matrix package was also 
included in version 4. Over the years, dozens of toolboxes (application libraries of specialized MATLAB files) 
have been added in areas such as signal processing, statistics, optimization, symbolic math, splines, and 
image processing. MATLAB’s matrix computations are now based on the LAPACK and BLAS software li- 
braries. MATLAB is widely used in linear algebra courses. Books such as [Leo06], [LHF03], and [HZ04] have 
extensive sets of MATLAB exercises and projects for the standard first course in linear algebra. The book by 
D.J. Higham and N.J. Higham [HH03] provides a comprehensive guide to all the basic features of MATLAB. 


71.1 Matrices, Submatrices, and Multidimensional Arrays 


Facts: 

1. The basic data elements that MATLAB uses are matrices. Matrices can be entered into a MATLAB 
session using square brackets. 

2. If A and B are matrices with the same number of rows, then one can append the columns of B to 
the matrix A and form an augmented matrix C by setting C = [A, B]. If E is a matrix with the 
same number of columns as A, then one can append the rows of E to A and form an augmented 
matrix F by setting F = [A; E]. 
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Row vectors of evenly spaced entries can be generated using MATLAB’s : operator. 
A submatrix of a matrix A is specified by A(u, v) where u and v are vectors that specify the row and 
column indices of the submatrix. 


. MATLAB arrays can have more than two dimensions. 


Commands: 


1. The number of rows and columns of a matrix can be determined using MATLAB’s size command. 
2. The command length(x) can be used to determine the number of entries in a vector x. The length 


command is equivalent to the command max(size(x)). 

MATLAB’s cat command can be used to concatenate two or more matrices along a single dimension 
and it can also be used to create multidimensional arrays. If two matrices A and B have the same 
number of columns, then the command cat(1, A, B) produces the same matrix as the command 
[A; B]. If the matrices B and C have the same number of rows, the command cat(2, B, C) produces 
the same matrix as the command [B, C]. The cat command can be used with more than two matrices 
as arguments. For example, the command cat(2, A, B, C, D) generates the same matrix as the com- 
mand [A, B, C, D]. IfA1, A2,..., Ak are m x n matrices, the command S = cat(3, Al, A2, ..., Ak) 
will generate an m x n x k array S. 


Examples: 


Ts 


The matrix 


is generated using the command 
A= it 24-3 
3.5 7 
7 a ae 


Alternatively, one could generate the matrix on a single line using semicolons to designate the 
ends of the rows 


A=[(1243;3572;241 1]. 


The command size(A) will return the vector (3,4) as the answer. The command size(A,1) will 
return the value 3, the number of rows of A, and the command size(A,2) will return the value 4, 
the number of columns of A. 

The command 


x=3:7 


will generate the vector x = (3, 4, 5, 6, 7). To change the step size to i, set 


z=3:0.5:7. 


This will generate the vectorz = (3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7). The MATLAB commands length(x) 
and length(z) will generate the answers 5 and 9, respectively. 


If A is the matrix in Example 1, then the command A(2,:) will generate the second row vector of 
A and the command A(:,3) will generate the third column vector of A. The submatrix of elements 
that are in the first two rows and last two columns is given by A(1:2, 3:4). Actually there is no need 
to use adjacent rows or columns. The command C = A([1 3], [1 3 4]) will generate the submatrix 


1 4 3 
CS 
E 1 | 
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4. The command 


S = cat(3, [512; 321], [123; 456]) 


will produce a2 x 3 x 2 array S with 


Seis). c= 
5 Dt. 22 
3 52s ol 
and 
S32) = 
1, 286e3 
4 5 6 


71.2 Matrix Arithmetic 


The six basic MATLAB operators for doing matrix arithmetic are: +, —, *, “, \, and /. The matrix left and 
right divide operators, \ and /, are described in Section 71.5. These same operators are also used for doing 
scalar arithmetic. 


Facts: 


1. If A and B are matrices with the same dimensions, then their sum and difference are computed 
using the commands: A + B and A— B. 

2. If B and C are matrices and the multiplication BC is possible, then the product E = BC is 
computed using the command 


E=BxC. 


3. The kth power of a square matrix A is computed with the command A‘k. 

4. Scalars can be either real or complex numbers. A complex number such as 3 + 47 is entered in 
MATLAB as 3 + 4i. It can also be entered as 3 + 4 sqrt(—1) or by using the command complex(3, 4). 
If i is used as a variable and assigned a value, say i = 5, then MATLAB will assign the expression 
3+ 4 i the value 23; however, the expression 3 + 4i will still represent the complex number 3 + 47. 
In the case that 7 is used as a variable and assigned a numerical value, one should be careful to enter 
a complex number of the form a + i (where a real) as a + li. 

5. MATLAB will perform arithmetic operations element-wise when the operator is preceded bya period 
in the MATLAB command. 

6. The conjugate transpose of a matrix B is computed using the command B’. If the matrix B is real, 
then B’ will be equal to the transpose of B. If B has complex entries, then one can take its transpose 
without conjugating using the command B’. 


Commands: 


1. The inverse of a nonsingular matrix C is computed using the command inv(C). 
2. The determinant of a square matrix A is computed using the command det(A). 
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Examples: 
1. If 


1 2 5 1 
=| | and =| | 
3 4 2 3 


the commands A * B and A « B will generate the matrices 


71.3 Built-In MATLAB Functions 


The inv and det commands are examples of built-in MATLAB functions. Both functions have a single input 
and a single output. Thus, the command d = det(A) has the matrix A as its input argument and the 
scalar d as its output argument. A MATLAB function may have many input and output arguments. When 
a command of the form 


[Aj,..., Ax] = fname(B,,...,B,) (71.1) 
is used to call a function fname with input arguments B),..., B,, MATLAB will execute the function routine 
and return the values of the output arguments Aj,..., Ag. 


Facts: 


1. The number of allowable input and output arguments for a MATLAB function is defined by a 
function statement in the MATLAB file that defines the function. (See section 71.8.) The function 
may require some or all of its input arguments. A MATLAB command of the form (71.1) may be 
used with j output arguments where 0 < j < k. The MATLAB help facility describes the various 
input and output options for each of the MATLAB commands. 


Examples: 


1. The MATLAB function pi is used to generate the number z. This function is used with no input 
arguments. 

2. The MATLAB function kron has two input arguments. If A and B are matrices, then the command 
kron(A,B) computes the Kronecker product of A and B. Thus, if A = [1, 2; 3,4] and B = 
[1, 1; 1, 1], then the command K = kron(A, B) produces the matrix 


Kk. -= 


2. 2 
2 2 
4 4 
4 4 


OW OW ee 
OW BC ee 


and the command L = kron(B, A) produces the matrix 


| a 


Oo Re OW 
rdw FF NY 
WO Re Wo re 
rd FF NY 
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3. One can compute the QZ factorization (see section 71.6) for the generalized eigenvalue problem 
using a command 
[E,F,Q,Z] = qz(A,B) 


with two input arguments and four outputs. The input arguments are square matrices A and B 
and the outputs are quasitriangular matrices E and F and unitary matrices Q and Z such that 


QAZ=E and QBZ=F. 
The command 
[E, F,Q, Z, V,W] = qz(A, B) 


will also compute matrices V and W of generalized eigenvectors. 


71.4 Special Matrices 


The ELMAT directory of MATLAB contains a collection of MATLAB functions for generating special types 
of matrices. 


Commands: 


1. The following table lists commands for generating various types of special matrices. 


Matrix Command Syntax Description 

eye eye(n) Identity matrix 

ones ones(n) or ones(m, n) Matrix whose entries are all equal to 1 
Zeros zeros(n) or zeros(m, n) Matrix whose entries are all equal to 0 
rand rand(n) or rand(m, n) Random matrix 

compan compan(p) Companion matrix 

hadamard —hadamard(n) Hadamard matrix 

gallery gallery(matname, pl, p2,...) Large collection of special test matrices 
hankel hankel(c) or hankel(c,r) Hankel matrix 

hilb hilb(n) Hilbert matrix 

invhilb invhilb(n) Inverse Hilbert matrix 

magic magic(n) Magic square 

pascal pascal(n) Pascal matrix 

rosser rosser Test matrix for eigenvalue solvers 
toeplitz toeplitz(c) or toeplitz(c,r) Toeplitz matrix 

vander vander(x) Vandermonde matrix 

wilkinson — wilkinson(n) Wilkinson’s eigenvalue test matrix 


2. The command gallery can be used to access a large collection of test matrices developed by 
N. J. Higham. Enter help gallery to obtain a list of all classes of gallery test matrices. 


Examples: 


1. The command rand(n) will generate an n x n matrix whose entries are random numbers that are 
uniformly distributed in the interval (0, 1). The command may be used with two input arguments 
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to generate nonsquare matrices. For example, the command rand(3,2) will generate a random 3 x 2 
matrix. The command rand when used by itself with no input arguments will generate a single 
random number between 0 and 1. 


. The command 


A = [eye(2), ones(2, 3); zeros(2), 2 * ones(2, 3) | 


will generate the matrix 


NHN NY FF 
Bo NO eS 
BO NO FR eR 


1 0 
0 1 
0 0 
0 0 


The command toeplitz(c) will generate a symmetric toeplitz matrix whose first column is the vector 
c. Thus, the command 


toeplitz([1; 2; 3]) 


will generate 


1 
2 
6) 


wo FF NN 
Pe NY Ww 


Note that in this case, since the toeplitz command was used with no output argument, the computed 
value of the command toeplitz(c) was assigned to the temporary variable ans. Further computations 
may end up overwriting the value of ans. To keep the matrix for further use in the MATLAB session, 
it is advisable to include an output argument in the calling statement. 

For anonsymmetric Toeplitz matrix it is necessary to include a second input argument r to define 
the first row of the matrix. If r(1) 4 c(1), the value of c(1) is used for the main diagonal. Thus, 
commands 


c= [1; 2; 3], r= [9,5,7], T= toeplitz(c, r) 


will generate 


1 
2 
3 


oF Ww 


7 
5 
1. 


The Toeplitz matrix generated is stored using the variable T. 
One of the classes of gallery test matrices is circulant matrices. These are generated using the MATLAB 
function circul. To see how to use this function, enter the command 


help private \ circul. 


The help information will tell you that the circul function requires an input vector v and that the 
command 


C = gallery(‘circul’, v) 
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will generate a circulant matrix whose first row is v. Thus, the command 
C = gallery(‘circul’, [4,5,6]) 


will generate the matrix 


an nN & 
Nn ek Ww 
ek UO 


71.5 Linear Systems and Least Squares 


The simplest way to solve a linear system in MATLAB is to use the matrix left divide operator. 


Facts: 


1. The symbol \ represents MATLAB’s matrix left divide operator. One can compute the solution to a 
linear system Ax = b by setting 


x= A\b. 


If A is ann x n matrix, then MATLAB will compute the solution using Gaussian elimination with 
partial pivoting. A warning message is given when the matrix is badly scaled or nearly singular. If 
the coefficient matrix is nonsquare, then MATLAB will return a least squares solution to the system 
that is essentially equivalent to computing A‘b (where A‘ denotes the pseudoinverse of A). In this 
case, MATLAB determines the numerical rank of the coefficient matrix using a QR decomposition 
and gives a warning when the matrix is rank deficient. 

If A isan m X n matrix and B is m x k, then the command 


C=A\B 
will produce an n x k matrix whose column vectors satisfy 
cj =A\bj j=1,...,k. 
2. The symbol / represents MATLAB’s matrix right divide operator. It is defined by 
B/A=(A'\B. 


In the case that A is nonsingular, the computation B/A is essentially the same as computing B A“, 
however, the computation is carried out without actually computing Aq. 


Commands: 


The following table lists some of the main MATLAB commands that are useful for linear systems. 
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Command Syntax 
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Description 


rref 

lu 
linsolve 
chol 
norm 
null 
null 
orth 
rank 
cond 


rcond 


Examples: 


U = rref(A) 
[L,U]=lu(A) 

x = linsolve(A , b , opts) 
R =chol(A) 

p = norm(X) 
U=null(A) 
R=null(A, ’r’) 

Q =orth(A) 
r=rank(A) 
c=cond(A) 
c=rcond(A) 
[Q,R] =qr(A) 
s=svd(A) 
[U,S,V] =svd(A) 
B= pinv(A) 


Reduced row echelon form of a matrix 

LU factorization 

Efficient solver for structured linear systems 
Cholesky factorization of a matrix 

Norm of a matrix or a vector 

Basis for the null space of a matrix 

Basis for null space rational form 

Orthonormal basis for the column space of a matrix 
Numerical rank of a matrix 

2-norm condition number for solving linear systems 
Reciprocal of approximate 1-norm condition number 
QR factorization 

Singular values of a matrix 

Singular value decomposition 


Pseudoinverse of a matrix 


1. The null command can be used to produce an orthonormal basis for the nullspace of a matrix. It 


can also be used to produce a “rational” nullspace basis obtained from the reduced row echelon 
form of the matrix. If 


diy “she ae 


yO (et Gr me Cd Ud Bi 


1 es (Soe Ce 


then the command U = null(A) will produce the matrix 


e— 


—0.8165 —0.0000 
0.4082 0.7071 
0.4082 —0.7071 

—0.0000 0.0000 


where the entries of U are shown in MATLAB’s format short (with four-digit mantissas). The column 
vectors of U form an orthonormal basis for the nullspace of A. The command R = null(A, 'r’) will 
produce a matrix R whose columns form a simple basis for the nullspace. 


R= 
—-1 -l 
1 0 
0 1 
0. 


2. MATLAB defines the numerical rank of a matrix to the number of singular values of the matrix that 
are greater than 


max(size(A)) * norm(A) * eps 
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where eps has the value 2°”, which is a measure of the precision used in MATLAB computations. 
Let H be the 12 x 12 Hilbert matrix. The singular values of H can be computed using the command 
s = svd(H). The smallest singular values are s(11) * 2.65 x 1074 and s(12) * 107'*. Since the 
value of eps is approximately 2.22 x 10~!°, the computed value of rank(H) will be the numerical 
rank 11, even though the exact rank of H is 12. The computed value of cond(H) is approximately 
1.8 x 10'° and the computed value of rcond(H) is approximately 2.6 x 107!7, 


71.6 Eigenvalues and Eigenvectors 


MATLAB’s eig function can be used to compute both the eigenvalues and eigenvectors of a matrix. 


Commands: 


1. The eig command. Given a square matrix A, the command e = eig(A) will generate a column 
vector e whose entries are the eigenvalues of A. The command [ X, D] = eig(A) will generate a 
matrix X whose column vectors are the eigenvectors of A and a diagonal matrix D whose diagonal 
entries are the eigenvalues of A. 

2. The eigshow command. MATLAB’s eigshow utility provides a visual demonstration of eigenvalues 
and eigenvectors of 2 x 2 matrices. The utility is invoked by the command eigshow(A). The input 
argument A must be a 2 x 2 matrix. The command can also be used with no input argument, in 
which case MATLAB will take [1 3; 4 2]/4 as the default 2 x 2 matrix. The eigshow utility shows 
how the image Ax changes as we rotate a unit vector x around a circle. This rotation is carried 
out manually using a mouse. If A has real eigenvalues, then we can observe the eigenvectors of the 
matrix when the vectors x and Ax are in the same or opposite directions. 

3. The command J = jordan(A) can be used to compute the Jordan canonical form of a matrix A. 
This command will only give accurate results if the entries of A are exactly represented, i.e., the 
entries must be integers or ratios of small integers. The command [X, J] = jordan(A) will also 
compute the similarity matrix X so that A= XJ X ear 

4. The following table lists some additional MATLAB functions that are useful for eigenvalue related 


problems. 
Function Command Syntax Description 
poly p= poly(A) Characteristic polynomial of a matrix 
hess H= hess(A) or [U, H] = hess(A) Hessenberg form 
schur T=schur(A) or[U,T]=schur(A) Schur decomposition 
qz [ E,F,Q,Z ]=qz(A,B) QZ factorization for generalized eigenvalues 
condeig s = condeig(A) Condition numbers for the eigenvalues of A 
expm E=expm(A) Matrix exponential 


71.7 Sparse Matrices 


A matrix is sparse if most of its entries are zero. MATLAB has a special data structure for handling sparse 
matrices. This structure stores the nonzero entries of a sparse matrix together with their row and column 
indices. 


Commands: 


1. The command sparse is used to generate sparse matrices. When used with a single input argument 
the command 


S = sparse(A) 
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will convert an ordinary MATLAB matrix A into a matrix S having the sparse data structure. More 
generally, a command of the form 


S = sparse(i, j, s, m,n, nzmax) 


will generate an m x n sparse matrix S whose nonzero entries are the entries of the vector s. The row 
and column indices of the nonzero entries are given by the vectors i and j. The last input argument 
nzmax specifies the total amount of space allocated for nonzero entries. If the allocation argument 
is omitted, by default MATLAB will set it to equal the value of length(s). 

2. MATLAB’s spy command can be used to plot the sparsity pattern of a matrix. In these plots the 
matrix is represented by a rectangular box with dots corresponding to the positions of its nonzero 
entries. 

3. The MATLAB directory SPARFUN contains a large collection of MATLAB functions for working with 
sparse matrices. The general sparse linear algebra functions are given in the following table. 


MATLAB Function Description 

eigs A few eigenvalues, using ARPACK 
svds A few singular values, using eigs 
luinc Incomplete LU factorization 
choline Incomplete Cholesky factorization 
normest Estimate the matrix 2-norm 
condest 1-norm condition number estimate 
sprank Structural rank 


All of these functions require a sparse matrix as an input argument. All have one basic output 
argument except in the case of luinc, where the basic output consists of the L and U factors. 

4. The SPARFUN directory also includes a collection of routines for the iterative solution of sparse 
linear systems. 


MATLAB Function Description 

pcg Preconditioned Conjugate Gradients Method 
bicg BiConjugate Gradients Method 

bicgstab BiConjugate Gradients Stabilized Method 

cgs Conjugate Gradients Squared Method 

gmres Generalized Minimum Residual Method 

Isqr Conjugate Gradients on the Normal Equations 
minres Minimum Residual Method 

qmr Quasi-Minimal Residual Method 

symmlq Symmetric LQ Method 


If A is a sparse coefficient matrix and B is a matrix of right-hand sides, then one can solve the equation 
AX = B using a command of the form X = fname(A,B), where fname is one of the iterative solver 
functions in the table. 


Examples: 


1. The command 
S = sparse([25, 37, 8], [211, 15, 92], [4.5, 3.2, 5.7], 200, 300) 
will generate a 200 x 300 sparse matrix S whose only nonzero entries are 


$25,211 = 4.5, $37,15 = 3.2, $892 = 5.7. 
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FIGURE71.1  Spy(B). 


The command B = bucky will generate the 60 x 60 sparse adjacency matrix B of the connectivity 
graph of the Buckminster Fuller geodesic dome and the command spy(B) will generate the spy plot 
shown in Figure 71.1. 


Programming 


MATLAB has built in all of the main structures one would expect from a high-level computer language. 
The user can extend MATLAB by adding on programs and new functions. 


Facts: 


1. MATLAB programs are called M-files and should be saved with a .m extension. 


. MATLAB programs may be in the form of script files that list a series of commands to be executed 


when the file is called in a MATLAB session, or they can be in the form of MATLAB functions. 


. MATLAB programs frequently include for loops, while loops, and if statements. 
. A function file must start with a function statement of the form 


function [oargl,...,oargk ] = fname(iargl, ..., iargj) 


where fname is the name of the function, iarg],. . . ,iargj are its input arguments, and oargl,. .. ,oargk 
are the output arguments. In calling a MATLAB function, it is not necessary to use all of the input and 
output allowed for in the general syntax of the command. In fact, MATLAB functions are commonly 
used with no output arguments whatsoever. 


. One can construct simple functions interactively in a MATLAB session using MATLAB’s inline com- 


mand. A simple function such as f(t) = t? + 4 can be described by the character array (or string) 
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“1? 4.4” The inline command will transform the string into a function for use in the current MATLAB 
session. Inline functions are particularly useful for creating functions that are used as input argu- 
ments for other MATLAB functions. An inline function is not saved as an m-file and consequently 
is lost when the MATLAB session is ended. 

6. One can use the same MATLAB command with varying amounts of input and output arguments. 
MATLAB keeps track of the number of input and output arguments included in the call statement 
using the functions nargin (the number of input arguments) and nargout (the number of out- 
put arguments). These commands are used inside the body of a MATLAB function to tailor the 
computations and output to the specifications of the calling statement. 

7. MATLAB has six relational operators that are used for comparisons of scalars or elementwise com- 
parisons of arrays. These operators are: 


Relational Operators 


< less than 

<= ___ less than or equal 

> greater than 

>= greater than or equal 
== equal 

~= not equal 


8. There are three logical operators as shown in the following table: 


Logical Operators 


& AND 
| OR 
~ NOT 


These logical operators regard any nonzero scalar as corresponding to TRUE and 0 as corresponding 
to FALSE. The operator & corresponds to the logical AND. If a and b are scalars, the expression 
a&b will equal 1 ifa and b are both nonzero (TRUE) and 0 otherwise. The operator | corresponds 
to the logical OR. The expression a|b will have the value 0 if a and b are both 0 and otherwise it 
will be equal to 1. The operator ~ corresponds to the logical NOT. For a scalar a, the expression 
~ a takes on the value 1 (TRUE) if a = 0 (FALSE) and the value 0 (FALSE) if a 4 0 (TRUE). 

For matrices these operators are applied element-wise. Thus, if A and B are both m x n matrices, 
then A&B is a matrix of zeros and ones whose (i, j) entry is a(i, j)&b(i, j). 


Examples: 


1. Given two m x n matrices A and B, the command C = A <B will generate an m x n matrix 
consisting of zeros and ones. The (i, j) entry will be equal to 1 if and only if a; < bj. If 


—1 1 0 
A= 42: 5]; 
T? 3% 2 


then command A >= 0 will generate 


ans = 


ee FS CO 
ooo OF 
oO _ 
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2 


If 
3 0 2 
— and B= F 
0 0 3 
then 
0 0 1 1 0 1 
A&B = » A|B= x AS ‘ 
0 1 0 1 1 0 
To construct the function g(t) = 3 cost — 2 sint interactively set 


g = inline(’3 * cos(t) — 2 * sin(t)’). 


If one then enters g(0) on the command line, MATLAB will return the answer 3. The command 
ezplot(g) will produce a plot of the graph of g(t). (See Section 71.9 for more information on 
producing graphics.) 

If the numerical nullity of a matrix is defined to be the number of columns of the matrix minus 
the numerical rank of the matrix, then one can create a file numnull.m to compute the numerical 
nullity of a matrix. This can be done using the following lines of code. 


function k = numnull(A) 
% The command numnull(A) computes the numerical nullity of A 
[m,n] = size(A); 
k =n -— rank(A); 
The line beginning with the % is a comment that is not executed. It will be displayed when the 


command help numnull is executed. The semicolons suppress the printouts of the individual com- 
putations that are performed in the function program. 


. The following is an example of a MATLAB function to compute the circle that gives the best least 


squares fit to a collection of points in the plane. 


function [center,radius,e] = circfit(x,y,w) 
% The command |[center,radius] = circfit(x,y) generates 
% the center and radius of the circle that gives the 
% best least squares fit to the data points specified 
% by the input vectors x and y. If a third input 
% argument is specified then the circle and data 
% points will be plotted. Specify a third output 
% argument to get an error vector showing how much 
% each point deviates from the circle. 
if size(x,1) == 1 & size(y,1) == 1 
Max = Ys 
end 
A = [2*x, 2*y, ones(size(x))]; 
b=x.%2+y.%2; 
c = A\b; 
center = c(1:2)’; 
radius = sqrt(c(3) + c(1)*2 + c(2)*2); 
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if nargin > 2 
t = 0:0.1:6.3; 


u = c(1) + radius « cos(t); 


v = c(2) + radius x sin(t); 
plot(x,y,’x’,u,v) 


axis(‘equal’) 


end 
if nargout == 3 

e = sqrt((x —c(1)).*2 + (y — ¢(2)).*2) — radius; 
end 


The command plot(x,y,’x’,u,v) is used to plot the original (x, y) data as discrete points in the plane, 


with each point designated by an “x,” and to also, on the same axis system, plot the (u, v) data points 
as a continuous curve. The following section explains MATLAB plot commands in greater detail. 


Graphics 


MATLAB graphics utilities allow the user to do simple two- and three-dimensional plots as well as more 
sophisticated graphical displays. 


Facts: 
iB 


2. 


3. 


4. 


MATLAB incorporates an objected-oriented graphics system called Handle Graphics. This system 
allows the user to modify and add on to existing figures and is useful in producing computer 
animations. 

MATLAB’s graphics capabilities include digital imaging tools. MATLAB images may be indexed or 
true color. 

An indexed image requires two matrices, a k x 3 colormap matrix whose rows are triples of numbers 
that specify red, green, blue intensities, and an m x n image matrix whose entries assign a colormap 
triple to each pixel of the image. 

A true color image is one derived from an m x n x 3 array, which specifies the red, green, blue 
triplets for each pixel of the image. 


Commands: 


lt, 


The plot command is used for simple plots of x-y data sets. Given a set of (x;, y;) data points, the 
command plot(x,y) plots the data points and by default sequentially draws line segments to connect 
the points. A third input argument may be used to specify a color (the default color for plots is 
black) or to specify a different form of plot such as discrete points or dashed line segments. 

The ezplot command is used for plots of functions. The command ezplot(f) plots the function 
f (x) on the default interval (—2z, 277) and the command ezplot(f,[a,b]) plots the function over 
the interval [a,b]. 

The commands plot3 and ezplot3 are used for three-dimensional plots. 

The command meshgrid is used to generate an xy-grid for surface and contour plots. Specifically 
the command [X,Y] = meshgrid(u, v) transforms the domain specified by vectors u and v into 
arrays X and Y that can be used for the evaluation of functions of two variables and 3-D surface 
plots. The rows of the output array X are copies of the vector u and the columns of the output array 
Y are copies of the vector v. The command can be used with only one input argument in which 
case meshgrid(u) will produce the same arrays as the command meshgrid(u,u). 
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5. The mesh command is used to produce wire frame surface plots and the command surf produces 


10. 


a solid surface plot. If [X,Y] = meshgrid(u,v) and Z(i,j) = f(ui,v;), then the command 
mesh(X,Y,Z) will producea wire frame plot of the function z = f(x, y) over the domain specified by 
the vectors u and v. Similarly the command surf(X,Y,Z) will generate a surface plot over the domain. 


. The MATLAB functions contour and ezcontour produce contour plots for functions of two variables. 
. The command meshc is used to graph both the mesh surface and the contour plot in the same graph- 


ics window. Similarly the command surfc will produce a surf plot with a contour graph appended. 
Given an array C whose entries are all real, the command image(C) will produce a two-dimensional 
image representation of the array. Each entry of C will correspond to a small patch of the image. The 
image array C may be either m x n or mx n x 3. IfC isan m x n matrix, then the colors assigned to 
each patch are determined by MATLAB’s current colormap. If C ism x n x 3, a true color array, then 
no color map is used. In this case the entries of C(:,:,1) determine the red intensities of the image, the 
entries of C(:,:,2) determine green intensities, and the elements of C(:,:,3) define the blue intensities. 
The colormap command is used to specify the current colormap for image plots of m x n arrays. 
The imread command is used to translate a standard graphics file, such as a gif, jpeg, or tiff file, 
into a true color array. The command can also be used with two output arguments to determine 
an indexed image representation of the graphics file. 


Examples: 


1. 


The graph of the function f(x) = cos(x) + sin?(x) on the interval (—2z, 27r) can be generated in 
MATLAB using the following commands: 


x = —6.3: 0.1:6.3; 
y = cos(x) + sin(x).%2; 
plot(x,y) 
The graph can also be generated using the ezplot command. (See Figure 71.2.) 
f = inline(‘cos(x) + sin(x).*2’) 
ezplot(f) 
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FIGURE 71.3 


2. We can generate a three-dimensional plot using the following commands: 


t = 0: pi/50: 10 x pi; 
plot3(t.*2. * sin(5 *t), t.*2. * cos(5 * t), t) 
grid on 


axis square 


These commands generate the plot shown in Figure 71.3. 
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3. 


MATLAB’s peaks function is a function of two variables obtained by translating and scaling Gaussian 
distributions. The commands 


[X,Y] = meshgrid(—3:0.1:3); 
Z = peaks(X,Y); 
meshc(X,Y,Z); 


generate the mesh and contour plots of the peaks function. (See Figure 71.4.) 


71.10 Symbolic Mathematics in MATLAB 


MATLAB’s Symbolic Toolbox is based upon the Maple kernel from the software package produced by 
Waterloo Maple, Inc. The toolbox allows users to do various types of symbolic computations using the 
MATLAB interface and standard MATLAB commands. All symbolic computations in MATLAB are performed 
by the Maple kernel. For details of how symbolic linear algebra computations such as matrix inverses and 
eigenvalues are carried out see Chapter 72. 


Facts: 


I; 


MATLAB’s symbolic toolbox allows the user to define a new data type, a symbolic object. The user 
can create symbolic variables and symbolic matrices (arrays containing symbolic variables). 

The standard matrix operations +, —, *, *,’ all work for symbolic matrices and also for combinations 
of symbolic and numeric matrices. To add a symbolic matrix and a numeric matrix, MATLAB first 
transforms the numeric matrix into a symbolic object and then performs the addition using the 
Maple kernel. The result will be a symbolic matrix. In general if the matrix operation involves at 
least one symbolic matrix, then the result will be a symbolic matrix. 

Standard MATLAB commands such as det, inv, eig, null, trace, rref, rank, and sum work for symbolic 
matrices. 

Not all of the MATLAB matrix commands work for symbolic matrices. Commands such as norm 
and orth do not work and none of the standard matrix factorizations such as LU or QR work. 
MATLAB’s symbolic toolbox supports variable precision floating arithmetic, which is carried out 
within the Maple kernel. 


Commands: 


I. 


SU RES 


The sym command can be used to transform any MATLAB data structure into a symbolic object. If 
the input argument is a string, the result is a symbolic number or variable. If the input argument 
is a numeric scalar or matrix, the result is a symbolic representation of the given numeric values. 
The syms command allows the user to create multiple symbolic variables with a single command. 
The command subs is used to substitute for variables in a symbolic expression. 

The command colspace is used to find a basis for the column space of a symbolic matrix. 

The commands ezplot, ezplot3, and ezsurf are used to plot symbolic functions of one or two 
variables. 

The command vpa(A,d) evaluates the matrix A using variable precision floating point arithmetic 
with d decimal digits of accuracy. The default value of d is 32, so if the second input argument is 
omitted, the matrix will be evaluated with 32 digits of accuracy. 


Examples: 


1. 


The command 


t = sym('t’) 
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transforms the string 't’ into a symbolic variable t. Once the symbolic variable has been defined, 
one can then perform symbolic operations. For example, the command 


factor(t*2 — 4) 
will result in the answer 
(t — 2) * (t+ 2). 
2. The command 
syms abc 
creates the symbolic variables a, b, and c. If we then set 
A=l[a, b, cb, c, ac, a, b] 


the result will be the symbolic matrix 


i = 
[a, b, c] 
[c, a, b] 
[b, c, a] 


Note that for a symbolic matrix the MATLAB output is in the form of a matrix of row vectors with 
each row vector enclosed by square brackets. 

3. Let A be the matrix defined in the previous example. We can add the 3 x 3 Hilbert matrix to A 
using the command B = A + hilb(3). The result is the symbolic matrix 


[atl, b+1/2, c+1/3] 
[c+1/4, a+1/5, b+1/6] 
[b+ 1/7, c+1/8, a+1/9]. 


To substitute 2 for a in the matrix A we set 
A = subs(A, a, 2). 


The matrix A then becomes 


= 
[2, b, c] 
[c, 2, b] 
[b, c, 2]. 


Multiple substitutions are also possible. To replace b by b + 1 and c by 5, one need only set 
A = subs(A,[b, c],[ b+ 1, 5]). 
4. Ifa is declared to be a symbolic variable, the command 


A=[121;242;00a] 
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will produce the symbolic matrix 


A = 
1, 2, 1] 
25 4, 6] 
0, 0, ai. 


The eigenvalues 0, 5, and a are computed using the command eig(A). The command 


[X, D] = eig(A) 


generates a symbolic matrix of eigenvectors 


i 1/2 *(a+8)/(—2+3x*a), 1] 
EE Ly 22) 
0, 1/2*xax(a—5)/(—2+3xa), 0] 


and the diagonal matrix 


D = 
[0, 0, 0] 
[0, a, 0] 
[0, 0, 5] 


When a = 0 the matrix A will be defective. One can substitute 0 for a in the matrix of eigenvectors 
using the command 


X = subs(X, a, 0). 


This produces the numeric matrix 


xX = 
2; 2. ~ 1 
1 Le +2: 
0 0 0 


5. Ifwesetz = exp(1), then MATLAB will compute an approximation to e that is accurate to 16 decimal 
digits. The command vpa(z) will produce a 32 digit representation of z, but only the first 16 digits 
will be accurate approximations to the digits of e. To compute e more accurately one should apply 
the vpa function to the symbolic expression 'exp(1)'. The command z = vpa(‘exp(1)’) produces an 
answer z = 2.71828182845904523536028747 13527, which is accurate to 32 digits. 


71.11 Graphical User Interfaces 


A graphical user interface (GUI) is a user interface whose components are graphical objects such as 
pushbuttons, radio buttons, text fields, sliders, checkboxes, and menus. These interfaces allow users to 
perform sophisticated computations and plots by simply typing numbers into boxes, clicking on buttons, 
or by moving slidebars. 
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FIGURE 71.5 Eigtool GUI. 


Commands: 


1. The command guide opens up the MATLAB GUI Design Environment. This environment is essen- 
tially a GUI containing tools to facilitate the creation of new GUIs. 


Examples: 


1. Thomas G. Wright of Oxford University has developed a MATLAB GUI, eigtool, for computing 
eigenvalues, pseudospectra, and related quantities for nonsymmetric matrices, both dense and 
sparse. It allows the user to graphically visualize the pseudospectra and field of values of a matrix 
with just the click of a button. 

The epsilon—pseudospectrum of a square matrix A is defined by 


A.(A) ={z€C|z¢€o(A+E) for some E with ||E|| < €}. (71.2) 


In Figure 71.5 the eigtool GUI is used to plot the epsilon—pseudospectra of a 10 x 10 matrix for 
FS 10 k= 0, L,9,3,4, 
For further information, see Chapter 16 and also references [Tre99] and [WT01]. 

2. The NSF-sponsored ATLAST Project has developed a large collection of MATLAB exercises, projects, 
and M-files for use in elementary linear algebra classes. (See [LHF03].) The ATLAST M-file collec- 
tion contains a number of programs that make use of MATLAB’s graphical user interface features to 
present user friendly tools for visualizing linear algebra. One example is the ATLAST cogame utility 
where students play a game to find linear combinations of two given vectors with the objective of 
obtaining a third vector that terminates at a given target point in the plane. Students can play the 
game at any one of four levels or play it competitively by selecting the two person game option. 
(See Figure 71.6.) At each step of the game a player must enter a pair of coordinates. MATLAB then 
plots the corresponding linear combination as a directed line segment. The game terminates when 
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FIGURE 71.6 ATLAST Coordinate Game. 
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FIGURE 71.7 ATLAST Transformation Utility. 
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the tip of the plotted line segment lies in the small target circle. A running list of the coordinates 
entered in the game is displayed in the lower box to the left of the figure. The cogame GUI is useful 
for teaching lessons on the span of vectors in R? and for teaching about different bases for R?. 

3. The ATLAST transform GUI helps students to visualize the effect of linear transformations on 
figures in the plane. With this utility students choose an image from a list of figures and then apply 
various transformations to the image. Each time a transformation is applied, the resulting image is 
shown in the current image window. The user can then click on the current transformation button 
to see the matrix representation of the transformation that maps the original image into the current 
image. In Figure 71.7 two transformations were applied to an initial image. First a 45° rotation was 
applied. Next a transformation matrix [1,0; 0.5, 1] was entered into the “Your Transformation” 
text field and the corresponding transformation was applied to the lower left image with the result 
being displayed in the Current Image window on the lower right. To transform the Current Image 
into the Target Image directly above it, one would need to apply a reflection transformation. 
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72.1 Introduction 


Maple® is a general purpose computational system, which combines symbolic computation with exact 
and approximate (floating-point) numerical computation and offers a comprehensive suite of scientific 
graphics as well. The main library of functions is written in the Maple programming language, a rich 
language designed to allow easy access to advanced mathematical algorithms. A special feature of Maple 
is user access to the source code for the library, including the ability to trace Maple’s execution and see its 
internal workings; only the parts of Maple that are compiled, for example, the kernel, cannot be traced. 
Another feature is that users can link to LAPACK library routines transparently, and thereby benefit from 
fast and reliable floating-point computation. The development of Maple started in the early 80s, and the 
company Maplesoft was founded in 1988. A strategic partnership with NAG Inc. in 2000 brought highly 
efficient numerical routines to Maple, including LAPACK. 

There are two linear algebra packages in Maple: LinearAlgebra and 1linalg. The linalg package 
is older and considered obsolete; it was replaced by LinearAlgebra in MAPLE 6. Here we describe only 
the LinearAlgebra package. The reader should be careful when reading other reference books, or 
the Maple help pages, to check whether reference is made to vector, matrix, array (notice the 
lower-case initial letter), which means that the older package is being discussed, ortoVector, Matrix, 
Array (with an upper-case initial letter), which means that the newer package is being discussed. 
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. Maple commands are typed after a prompt symbol, which by default is “greater than” ( > ). In 


examples below, keyboard input is simulated by prefixing the actual command typed with the 
prompt symbol. 
In the examples below, some of the commands are too long to fit on one line. In such cases, the 
Maple continuation character backslash ( \ ) is used to break the command across a line. 
Maple commands are terminated by either semicolon ( ; ) or colon (: ). Before Maple 10, a 
terminator was required, but in the Maple 10 GUI it can be replaced by a carriage return. The 
semicolon terminator allows the output of a command to be displayed, while the colon suppresses 
the display (but the command still executes). 
To access the commands described below, load the LinearAlgebra package by typing the 
command (after the prompt, as shown) 
> with( LinearAlgebra ); 

If the package is not loaded, then either a typed command will not be recognized, or a different 
command with the same name will be used. 


. The results of a command can be assigned to one or more variables. Thus, 


> ase 1 ; 

assigns the value 1 to the variable a, while 

> (a,b,c) = 1,2,3 ; 

assigns a the value 1, b the value 2 and c the value 3. Caution: The operator colon-equals ( := ) 
is assignment, while the operator equals ( = ) defines an equation with a left-hand side and a 
right-hand side. 


. A sequence of expressions separated by commas is an expression sequence in Maple, and some 


commands return expression sequences, which can be assigned as above. 
Ranges in Maple are generally defined using a pair of periods ( .. ). The rules for the ranges of 
subscripts are given below. 


Vectors 


Facts: 


i 


In Maple, vectors are not just lists of elements. Maple separates the idea of the mathematical object 
Vector from the data object Array (see Section 72.4). 


. A Maple Vector can be converted to an Array, and an Array of appropriate shape can be 


converted to a Vector, but they cannot be used interchangeably in commands. See the help file 
for convert to find out about other conversions. 

Maple distinguishes between column vectors, the default, and row vectors. The two types of vectors 
behave differently, and are not merely presentational alternatives. 


Commands: 


1. 


Generation of vectors: 


* Vector( [x),X2,...] ) Construct a column vector by listing its elements. The length of the 
list specifies the dimension. 


* Vector[column] ( [x,%,... ] ) Explicitly declare the column attribute. 
* Vector[row] ( [x,,x2,... ]  ) Construct a row vector by initializing its elements from a 
list. 


* <v),V2,...> Construct a column vector with elements v;, v2, etc. An element can be another 
column vector. 


Linear Algebra in Maple 72-3 


* <v,|v2|... > Construct a row vector with elements v, v2, etc. An element can be another row 
vector. A useful mnemonic is that the vertical bars remind us of the column dividers in a table. 


* Vector( n, k—>f£(k) ). Construct an n-dimensional vector using a function f(k) to 
define the elements. f(k) is evaluated sequentially for k from 1 to n. The notation k—>f (k) is 
Maple syntax for a univariate function. 


* Vector( n, fill=v ) Ann-dimensional vector with every element v. 
* Vector( n, symbol=v ) Ann-dimensional vector containing symbolic components vx. 
* map( x—>f(x), V_) Construct anew vector by applying function f(x) to each element of 
the vector named V. Caution: the command is map not Map. 
2. Operations and functions: 
* v[i] Elementi of vector v. The result is a scalar. Caution: A symbolic reference v [i] is typeset 
as vj on output in a Maple worksheet. 


* vip. .q] Vector consisting of elements vj, p < i < q. The result is a Vector, even for the 
case v[p. .p]. Either of p or q can be negative, meaning that the location is found by counting 
backwards from the end of the vector, with —1 being the last element. 


* ut+v, u-v Add or subtract Vectors u, v. 


* axv Multiply vector v by scalar a. Notice the operator is “asterisk” (+). 


*u . v,DotProduct( u, v_ ) The inner product of Vectors u and v. See examples for 
complex conjugation rules. Notice the operator is “period” (.) not “asterisk” («) because inner 
product is not commutative over the field of complex numbers. 


* Transpose( v ), vA%T Change a column vector into a row vector, or vice versa. 
Complex elements are not conjugated. 


* HermitianTranspose( v ), VA%H Transpose with complex conjugation. 


* OuterProductMatrix( u, v_ ) The outer product of Vectors uw and v (ignoring the 
row/column attribute). 


* CrossProduct( u, v ), u &x v The vector product, or cross product, of three- 
dimensional vectors u, v. 


* Norm( v, 2 ) The 2-norm or Euclidean norm of vector v. Notice that the second argument, 
namely the 2, is necessary, because Norm( v_) defaults to the infinity norm, which is different 
from the default in many textbooks and software packages. 


* Norm( v, p_ ) The p-norm ofv, namely (37"_, |v;|?)"/?) . 


Examples: 


In this section, the imaginary unit is the Maple default J. That is, —1 = I. In the matrix section, we 
show how this can be changed. To save space, we shall mostly use row vectors in the examples. 


1. Generate vectors. The same vector created different ways. 
> Vector [row] ([0,3,8]): <0|3|8>: Transpose(<0,3,8>): Vector[row] 
(3,1->1A2-1); 


[0, 3, 8] 
2. Selecting elements. 
> V:s<alb|[c|dle|f>: Vl := V[2 .. 4]; V2:=v[-4 .. -1]; 
V3:=V[-4 .. 4]; 


V1:=[b,c,d], V2:=[c,d,e,f], V3:= [c,d] 
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3. A Gram-—Schmidt exercise. 


> ul := <3/0|4>: u2 := <2|1|1>: wln := ul/Norm( ul, 2 ); 
win := [3/5,0,4/5] 


> w2 := u2 - (u2 . win)*xwiln; w2n := w2/Norm( w2, 2 ); 


D2 xf Bx/2 
w2n := . ; 
5 2 10 


4. Vectors with complex elements. Define column vectors u,,v, and row vectors u,,V;. 


> uc = <1 + 1,2>: ur := Transpose( uc ): ve := <5,2 — 3*I>: 
vr := Transpose( vc ): 

The inner product of column vectors conjugates the first vector in the product, and the inner 
product of row vectors conjugates the second. 
> innerl := uc . vc; inner2 := ur. vr; 

innerl := 9-11 I, inner2 := 94+11 1 

Maple computes the product of two similar vectors, i.e., both rows or both columns, as a true 
mathematical inner product, since that is the only definition possible; in contrast, if the user mixes 
row and column vectors, then Maple does not conjugate: 
> but := ur . vc; 


but :=9-I1 


Caution: The use of a period (.) with complex row and column vectors together differs from the use of a 
period (.) with complex 1 x mand m x 1 matrices. In case of doubt, use matrices and conjugate explicitly 
where desired. 


72.3. Matrices 


Facts: 


1. 
2. 


One-column matrices and vectors are not interchangeable in Maple. 
Matrices and two-dimensional arrays are not interchangeable in Maple. 


Commands: 


1. 


Generation of Matrices. 


* Matrix( [[a,b,...],[c,d,...],...] ) Construct a matrix row-by-row, using a list of lists. 


* << alb|...>,<c|d|...>,...> Construct a matrix row-by-row using vectors. Notice that 
the rows are specified by row vectors, requiring the | notation. 


* << a,b,...>|< c,d,...>|...>Constructa matrix column-by-column using vectors. No- 
tice that each vector is a column, and the columns are joined using | , the column operator. 


Caution: Both variants of the << ... >> constructor are meant for interactive use, not pro- 
grammatic use. They are slow, especially for large matrices. 
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Matrix( n, m, (i,j3)—>f£(i,j) ) Construct a matrix n x m using a function f (i, j) 
to define the elements. f (i, j) is evaluated sequentially for i from 1 to n and j from 1 to m. The 
notation (i,j3)—>£(i,j) is Maple syntax for a bivariate function f (i,j). 


Matrix( n, m, fill=a ) Ann x m matrix with each element equal to a. 
Matrix( n, m, symbol=a ) Ann x m matrix containing subscripted entries Ajj. 
map( x—>f(x), M ) A matrix obtained by applying f(x) to each element of M. 


Caution: the command ismap not Map.]| 


<< A|B>, < C|D>> Construct a partitioned or block matrix from matrices A, B,C, D. 
Note that < A|B > will be formed by adjoining columns; the block < C|D > will be placed 
below < A|B >. The Maple syntax is similar to a common textbook notation for partitioned 
matrices. 


2. Operations and functions 


M[i,3] Elementi, j of matrix M. The result is a scalar. 
M[1..—1,k] Column k of Matrix M. The result isa Vector. 
M[{k,1..—1] Rowk of Matrix M. The result is a row Vector. 


M[p..q,xr..s] Matrix consisting of submatrix mjj, p < i < q,r < j < s. In HANDBOOK 
notation, M[{p,...,q}, {r,...5s}]. 


Transpose( M ), MAS%T Transpose matrix M, without taking the complex conjugate of 
the elements. 


HermitianTranspose( M ), MA%H Transpose matrix M, taking the complex conjugate 
of elements. 


A+B Add/subtract compatible matrices or vectors A, B. 


A.B Product of compatible matrices or vectors A, B. The examples below detail the ways in 
which Maple interprets products, since there are differences between Maple and other software 
packages. 


MatrixInverse( A ), AA(—1) Inverse of matrix A. 


Determinant ( A ) Determinant of matrix A. 


Norm( A, 2 ) The (subordinate) 2-norm of matrix A, namely maxy,,),, =1 || Au||2 where the 
norm in the definition is the vector 2-norm. 


Cautions: 


(a) Notice that the second argument, i.e., 2, is necessary because Norm( A_) defaults to the 
infinity norm, which is different from the default in many textbooks and software packages. 


(b) Notice also that this is the largest singular value of A, and is usually different from the Frobe- 
nius norm ||Al|z, accessed by Norm( A, Frobenius ), which is the 
Euclidean norm of the vector of elements of the matrix A. 


(c) Unless A has floating-point entries, this norm will not usually be computable explicitly, and 


it may be expensive even to try. 


Norm( A, p_ ) The (subordinate) matrix p-norm of A, for integers p >= 1 or for p being 
the symbol infinity, which is the default value. 
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Examples: 
1. A matrix product. 

> A: <<1|-2|3>,<0|1|1>>; B := Matrix(3, 2, symbol=b); C= A 

B; 
by by 

Rel ae B:=|by b 
= 0 1 1 5) —_ 21 22-13: 

bs, bsp 


Gk by, — 2b2; + 3b3; bg — 2by2 + 3b32 
ba; + bs, ba + b32 : 


2. A Gram-Schmidt calculation revisited. 
If uy, U2 are m x 1 column matrices, then the Gram—Schmidt process is often written in textbooks 
as 
T 


Uy Uy 


Uy). 
ul uy 


W2 = U2 — 


Notice, however, that ut u, and ul uy are strictly 1 x 1 matrices. Textbooks often skip over the 
conversion of uf u, froma 1 x 1 matrix toa scalar. Maple, in contrast, does not convert automatically. 

Transcribing the printed formula into Maple will cause an error. Here is the way to do it, reusing 
the earlier numerical data. 


> ul := <<3,0,4>>; u2 := <<2,1,1>>; r := u2*°S$T . ul; 
s := ul*sT . ul; 
3 2 
ul:= |]0O] , uw2:= ]1] , r:=[10], s := [25]. 
4 1 


Notice the brackets in the values of r and s because they are matrices. Since r [1,1] ands[1,1] 
are scalars, we write 

> w2 := u2 - r[{1,1]/s[1,1]*ul; 

and reobtain the result from Example 3 in Section 72.2. Alternatively, u1 and u2 can be converted 
to Vectors first and then used to form a proper scalar inner product. 

> res u2[l..-L1,1) . ulf[l..-1,2)] 7 Ss =] wh[l..—1, 0) .-uwl(l..—1,1); 
w2 := u2-r/s*ul; 


—3/5 


3. Vector—Matrix and Matrix—Vector products. 
Many textbooks equate a column vector and a one-column matrix, but this is not generally so in 
Maple. Thus 
> bre <1,2>; Bie <<1,2>>; C = <<4|5|6>>; 


s[fe-[enters 


Only the product B . C is defined, and the product b . C causes an error. 


Se Br og! 3 
4 5 6 
8 10 12° 
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The rules for mixed products are 


Vector[row] ( m ) 


Vector[row](n ) . Matrix( n, m ) 
) Vector[column](n ) 


Matrix( n, m Vector[column] ( m ) 


ThecombinationsVector(n). Matrix(1, m)and Matrix(m, 1). Vector[row] (n) 
cause errors. If users do not want this level of rigor, then the easiest thing to do is to use only the 
Matrix declaration. 

4. Working with matrices containing complex elements. 
First, notation: In linear algebra, I is commonly used for the identity matrix. This corresponds to 
the eye function in MaTLas. However, by default, Maple uses I for the imaginary unit, as seen 
in section 72.2. We can, however, use I for an identity matrix by changing the imaginary unit to 
something else, say _i. 
> interface( imaginaryunit=_i): 
As the saying goes: An _i for an I and an I for an eye. 

Now we can calculate eigenvalues using notation similar to introductory textbooks. 


> A := <<1,2>|<-2,1>>; I := IdentityMatrix( 2 ); 
p := Determinant ( x*I-A ); 
1 -2 1 0 : 
A:= ee > prex°—2x+5. 
2 1 0 1 


Solving p = 0, we obtain eigenvalues 1 + 2i, 1 — 2i. With the above setting of imaginaryunit, 
Maple will print these values as 1+2 _i, 1-2 _i, but we have translated back to standard 


mathematical i, where i = —1. 

5. Moore—Penrose inverse. Consider M := Matrix(3,2,[[1,1], l[a,aA2], [aA2,a]]);, 
a 3 x 2 matrix containing a symbolic parameter a. We compute its Moore—Penrose pseudoin- 
verse and a proviso guaranteeing correctness by the command > (Mi, p):= MatrixInverse\ 
(M, method=pseudo, output=[inverse, proviso]); which assigns the 2 x 3 


pseudoinverse to Mi and an expression, which if nonzero guarantees that Mi is the correct (unique) 
Moore-—Penrose pseudoinverse of M. Here we have 


(242 Biche gp Poe Ay aaa at+aetl 
Mi fee a(a>+a*—a?—a?+2a—2)  a(a>+a*t—a?—a*+2a—2) 
= 
-1 at*+ae+] atar+l 
2 2 =) 2 4 
( paca +a‘) a(a°>+at—a?—a?+2a—-2) a(a> +a‘ —a> —a*+2a—2) 
and p = a? — a. Thus, ifa 4 0 anda F 1, the computed pseudoinverse is correct. By separate 
computations we find that the pseudoinverse of M|,= is 
1/2 0 0 
1/2 0 0 
and that the pseudoinverse of M|,—1 is 
1/6 1/6 1/6 
1/6 1/6 1/6 


and moreover that these are not special cases of the generic answer returned previously. In a certain 
sense this is obvious: the Moore—Penrose inverse is discontinuous, even for square matrices (consider 
(A — A1)71, for example, as 2 — an eigenvalue of A). 
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72.4 Arrays 


Before describing Maple’s Array structure, it is useful to say why Maple distinguishes between an Array 
anda Vector or Matrix, when other books and software systems do not. In linear algebra, two different 
types of operations are performed with vectors or matrices. The first type is described in Sections 72.2 and 
72.3, and comprises operations derived from the mathematical structure of vector spaces. The other type 
comprises operations that treat vectors or matrices as data arrays; they manipulate the individual elements 
directly. As an example, consider dividing the elements of Array [1, 3,5] by the elements of [7, 11, 13] to 
obtain [1/7, 3/11, 5/13]. 

The distinction between the operations can be made in two places: in the name of the operation or the 
name of the object. In other words we can overload the data objects or overload the operators. Systems such 
as MATLAB choose to leave the data object unchanged, and define separate operators. Thus, in MATLAB the 
statements [1,3,5]/[7, 11, 13] and [1, 3, 5]./[7, 11, 13] are different because of the operators. In contrast, 
Maple chooses to make the distinction in the data object, as will now be described. 


Facts: 


1. The Maple Array is a general data structure akin to arrays in other programming languages. 
2. An array can have up to 63 indices and each index can lie in any integer range. 
3. The description here only addresses the overlap between Maple Array and Vector. 


Caution: A Maple Array might look the same as a vector or matrix when printed. 


Commands: 


1. Generation of arrays. 


* Array([x),%,...]) Construct an array by listing its elements. 

* Array( m..n ) Declare an empty 1-dimensional array indexed from m to n. 

* Array( v ) Use an existing Vector to generate an array. 

* convert( v, Array ) Convert a Vector v into an Array. Similarly, a Matrix 


can be converted to an Array. See the help file for rtable_options for advanced methods 
to convert efficiently, in-place. 


2. Operations (memory/stack limitations may restrict operations). 


‘aan Raise each element of a to power n. 


‘ax bat+bea-—b Multiply (add, subtract) elements of b by (to, from) elements 
of a. 


‘a/b Divide elements of a by elements of b. Division by zero will produce unde- 
fined or infinity (or exceptions can be caught by user-set traps; see the help file for 
Numeric_Events). 


Examples: 


1. Array arithmetic. 
> simplify( (Array([25,9,4])*Array(1..3,x->xA2-1 )/Array(<5,3,2>\ 
)JA(1/2)); 


[0, 3, 4] 
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2 


Getting Vectors and Arrays to do the same thing. 
> Transpose( map(x->x*x,<1,2,3> )) - convert(Array( [1,2,3] )A2,\ 
Vector) ; 


[0, 0, 0] 


72.5 Equation Solving and Matrix Factoring 


Cautions: 


I 


Ifa matrix contains exact numerical entries, typically integers or rationals, then the material studied 
in introductory textbooks transfers to a computer algebra system without special considerations. 
However, if a matrix contains symbolic entries, then the fact that computations are completed 
without the user seeing the intermediate steps can lead to unexpected results. 

Some of the most popular matrix functions are discontinuous when applied to matrices containing 
symbolic entries. Examples are given below. 


. Some algorithms taught to educate students about the concepts of linear algebra often turn out 


to be ill-advised in practice: computing the characteristic polynomial and then solving it to find 
eigenvalues, for example; using Gaussian elimination without pivoting on a matrix containing 
floating-point entries, for another. 


Commands: 
1. LinearSolve( A, B )_ The vector or matrix X satisfying AX = B. 
2. BackwardSubstitute( A, B ), ForwardSubstitute( A, B ) The vector or 
matrix X satisfying AX = B when A is upper or lower triangular (echelon) form, respectively. 
3. ReducedRowEchelonForm( A ). The reduced row-echelon form (RREF) of the matrix A. 
For matrices with symbolic entries, see the examples below for recommended usage. 
4. Rank( A ) Therankofthe matrix A. Caution: If A has floating-point entries, see the section 


9. 
10. 


below on Numerical Linear Algebra. On the other hand, if A contains symbolic entries, then the 
rank may change discontinuously and the generic answer returned by Rank may be incorrect for 
some specializations of the parameters. 

NullSpace( A )_ The nullspace (kernel) of the matrix A. Caution: If A has floating-point 
entries, see the section below on Numerical Linear Algebra. Again on the other hand, if A contains 
symbolic entries, the nullspace may change discontinuously and the generic answer returned by 
Null1Space may be incorrect for some specializations of the parameters. 


( P, L, U, R ) := LUDecomposition( A, method='RREF' ) ThePLUR,or 
Turing, factors of the matrix A. See examples for usage. 
( P, L, U ) := LUDecomposition( A ) The PLU factors of a matrix A, when the 


RREF R is not needed. This is usually the case for a Turing factoring where R is guaranteed (or 
known a priori) to be I, the identity matrix, for all values of the parameters. 

( Q, R ) := QRDecomposition( A, fullspan ) The QR factors of the matrix A. 
The option full span ensures that Q is square. 

SingularValues( A ) See Section 72.8, Numerical Linear Algebra. 
ConditionNumber( A ) See Section 72.8, Numerical Linear Algebra. 


Examples: 


1. 


Need for Turing factoring. 

One of the strengths of Maple is computation with symbolic quantities. When standard linear 
algebra methods are applied to matrices containing symbolic entries, the user must be aware of 
new mathematical features that can arise. The main feature is the discontinuity of standard matrix 
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functions, such as the reduced row-echelon form and the rank, both of which can be discontinuous. 


For example, the matrix 
7—-—ix 4 
B=A-AI= 
6 2-iX 


has the reduced row-echelon form 


0 0 


: ") poeee 
0 0 


Notice that the function is discontinuous precisely at the interesting values of 4. Computer algebra 
systems in general, and Maple in particular, return “generic” results. Thus, in Maple, we have 
> B:= << 7-x | 4 >, < 6 | 2-x >>; 


7—x 4 
B= ; 
6 2—-—x 


> ReducedRowEchelonForm( B ) 


T2473 
ReducedRowEchelonForm(B) = 4 = 10, 


This difficulty is discussed at length in [CJ92] and [CJ97]. The recommended solution is to use 
Turing factoring (generalized P LU decomposition) to obtain the reduced row-echelon form with 
provisos. Thus, for example, 

> A := <<1|-2|3|sin(x)>,<1|4*cos (x) |3|3*sin(x)>,<-1|3|cos(x)-3|\ 
COs (X)>>; 


1 —2 3 sin x 
A:=|]1 4cosx 3 3sinx 
—l 3 cosx—3 cosx 
> ( P, L, U, R ) := LUDecomposition( A, method='!RREF' ): 


The generic reduced row-echelon form is then given by 
1 0 O (2sinxcosx — 3sinx — 6cosx — 3)/(2cosx + 1) 
R=]0 1 0 sin x/(2cosx + 1) 
0 0 1 (2cosx +1+2sinx)/(2.cosx + 1) 


This shows a visible failure when 2 cosx + 1 = 0, but the other discontinuity is invisible, and 
requires the U factor from the Turing (P LU R) factors, 
1 —2 3 
U=1|0 4cosx+2 0 
0 0 cos x 


to see that the case cos x = 0 also causes failure. In both cases (meaning the cases 2 cosx + 1 = 0 
and cos x = 0), the RREF must be recomputed to obtain the singular cases correctly. 
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2. QR factoring. 
Maple does not offer column pivoting, so in pathological cases the factoring may not be unique, 
and will vary between software systems. For example, 
> A := <<0,0> | <5,12>>: QRDecomposition( A, fullspan ) 


5/13 12/13 0 13 
12/13 —5/13|° |o o| 


72.6 Eigenvalues and Eigenvectors 


Facts: 


1. In exact arithmetic, explicit expressions are not possible in general for the eigenvalues of a matrix 
of dimension 5 or higher. 

2. When it has to, Maple represents polynomial roots (and, hence, eigenvalues) implicitly by the 

RootOf construct. Expressions containing Root Ofs can be simplified and evaluated numerically. 


Commands: 
1. Eigenvalues( A )_ The eigenvalues of matrix A. 
2. Eigenvectors( A )_ The eigenvalues and corresponding eigenvectors of A. 
3. CharacteristicPolynomial( A, 'x' )  Thecharacteristicpolynomial of A expressed 


using the variable x. 
4. JordanForm( A )_ The Jordan form of the matrix A. 


Examples: 


1. Simple eigensystem computation. 
> Eigenvectors( <<7,6>|<4,2>> ); 


=i) 2) A738 
10]°| 1 1 | 


So the eigenvalues are —1 and 10 with the corresponding eigenvectors [—1/2, 1] and [4/3, 1]". 
2. A defective matrix. 

If the matrix is defective, then by convention the matrix of “eigenvectors” returned by Maple 

contains one or more columns of zeros. 

> Eigenvectors( <<1,0>|<1,1>> ); 


3. Larger systems. 
For larger matrices, the eigenvectors will use the Maple Root Of construction, 


3 1 7 1 
5 6 -—3 5 
A:= ; 
3 -l1 —-1 0 
—1 5 I. 25 
> (L, V ) := Eigenvectors( A ): The colon suppresses printing. The vector of 


eigenvalues is returned as 
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RootOf (_Z* — 13 _Z? — 4 _Z’ + 319 _Z — 386, index = 1) 
RootOf (_Z* — 13 _Z? — 4_Z? + 319 _Z — 386, index = 2) 
RootOf (_Z* — 13 _Z? — 4_Z? + 319 _Z — 386, index = 3) 
RootOf (_Z* — 13 _Z? —4_Z* + 319 Z — 386, index = 4) 


This, of course, simply reflects the characteristic polynomial: 
> CharacteristicPolynomial( A, 'x' ); 


x* — 13x? — 4x? + 319x — 386 


The Eigenvalues command solves a 4th degree characteristic polynomial explicitly in terms 
of radicals unless the option implicit is used. 
4. Jordan form. Caution: As with the reduced row-echelon form, the Jordan form ofa matrix containing 
symbolic elements can be discontinuous. For example, given 


> (J, Q ) := JordanForm( A, output=['J','Q'] ); 


1 1] fe o 
ax |i ele i 


with A = QJ Q7!. Note that Q is invertible precisely when t 4 0. This gives a proviso on the 
correctness of the result: J will be the Jordan form of A only for t 4 0, which we see is the generic 
case returned by Maple. 

Caution: Exact computation has its limitations, even without symbolic entries. If we ask for the 
Jordan form of the matrix 


—1 4 —1 —14 20 —8 
4 —5 —63 203. —217 78 
—l —63 403 —893 834 —280 
—14 203 —893 1703 —1469 470 
20 —217 834 —1469 1204 —372 
—8 78 —280 470 —372 112 


a relatively modest 6 x 6 matrix with a triple eigenvalue 0, then the transformation matrix Q as 
produced by Maple has entries over 35,000 characters long. Some scheme of compression or large 
expression management is thereby mandated. 


72.7 Linear Algebra with Modular Arithmetic in Maple 


There is a subpackage, LinearAlgebra [Modular], designed for programmatic use, that offers access 
to modular arithmetic with matrices and vectors. 


Facts: 


1. The subpackage can be loaded by issuing the command 
> with( LinearAlgebra[Modular] ) ; which gives access to the commands 


[AddMultiple, Adjoint, BackwardSubstitute, Basis, Characteristic 
Polynomial, ChineseRemainder, Copy, Create, Determinant, Fill, 
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ForwardSubstitute, Identity, Inverse, LUApply, LUDecomposition, 
LinIntSolve, MatBasis, MatGcd, Mod, Multiply, Permute, Random, 
Rank, RankProfile, RowEchelonTransform, RowReduce, Swap, 
Transpose, ZigZag] 


2. Arithmetic can be done modulo a prime p or, in some cases, a composite modulus m. 
3. The relevant matrix and vector datatypes are integer [4], integer [8], integer [], and 
float [8]. Use of the correct datatype can improve efficiency. 


Examples: 
Sp = 13; 
> A := Mod( p, Matrix([[1,2,3],[4,5,6],[7,8,-9]]), integer[4] ); 


> Mod( p, MatrixInverse( A ), integer[4] ); 


Cautions: 


1. This is not to be confused with the mod utilities, which together with the inert Inverse command, 
can also be used to calculate inverses in a modular way. 

2. One must always specify the datatype in Modular commands, or a cryptic error message will be 
generated. 


72.8 Numerical Linear Algebra in Maple 


The above sections have covered the use of Maple for exact computations of the types met during a 
standard first course on linear algebra. However, in addition to exact computation, Maple offers a variety 
of floating-point numerical linear algebra support. 


Facts: 


1. Maple can compute with either “hardware floats” or “software floats.” 
2. A hardware float is IEEE double precision, with a mantissa of (approximately) 15 decimal digits. 
3. A software float has a mantissa whose length is set by the Maple variable Digits. 


Cautions: 


1. If an integer is typed with a decimal point, then Maple treats it as a software float. 
2. Software floats are significantly slower that hardware floats, even for the same precision. 


Commands: 


l. Matrix( n, m, datatype=float[8] ) Ann x m matrix of hardware floats (initial- 
ization data not shown). The elements must be real numbers. The 8 refers to the number of bytes 
used to store the floating point real number. 

2. Matrix( n, m, datatype=complex(float[8] )) Ann x m matrix of hardware 
floats, including complex hardware floats. A complex hardware float takes two 8-byte storage 
locations. 
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3. Matrix( n, m, datatype=sfloat ) Ann x m matrix of software floats. The entries 
must be real and the precision is determined by the value of Digits. 

4. Matrix( n, m, datatype=complex(sfloat) ) As before with complex software 
floats. 

5. Matrix( n, m, shape=symmetric ) A matrix declared to be symmetric. Maple can 


take advantage of shape declarations such as this. 


Examples: 


1. A characteristic surprise. 
When asked to compute the characteristic polynomial of a floating-point matrix, Maple first com- 
putes eigenvalues (by a good numerical method) and then presents the characteristic polynomial 
in factored form, with good approximate roots. Thus, 
> CharacteristicPolynomial( Matrix( 2, 2, [[666,667],[665,666]],\ 
datatype=float[8]), 'x' ); 


(x — 1331.99924924882612 — 0.07) (x — 0.000750751173882235889 — 0.07) . 


Notice the signed zero in the imaginary part; though the roots in this case are real, approximate 
computation of the eigenvalues of a nonsymmetric matrix takes place over the complex numbers. 
(n.b.: The output above has been edited for clarity.) 

2. Symmetric matrices. 
If Maple knows that a matrix is symmetric, then it uses appropriate routines. Without the symmet- 
ric declaration, the calculation is 
> Eigenvalues( Matrix( 2, 2, [[1,3],[3,4]], datatype=float[8]) ); 


—0.854101966249684707 + 0.01 
5.85410196624968470 + 0.01 | 


With the declaration, the computation is 
> Eigenvalues( Matrix( 2, 2, [[1,3],[3,4]], shape=symmetric, 
datatype=float[8]) ); 


—0.854101966249684818 
5.85410196624968470 | 


Cautions: Use of the shape=symmetric declaration will force Maple to treat the matrix as being 
symmetric, even if it is not. 

3. Printing of hardware floats. 
Maple prints hardware floating-point data as 18-digit numbers. This does not imply that all 18 digits 
are correct; Maple prints the hardware floats this way so that a cycle of converting from binary to 
decimal and back to binary will return to exactly the starting binary floating-point number. Notice 
in the previous example that the last 3 digits differ between the two function calls. In fact, neither 
set of digits is correct, as a calculation in software floats with higher precision shows: 
> Digits := 30: Eigenvalues( Matrix( 2, 2, [[1,3],[3,4]l, \ 

datatype=sfloat, shape=symmetric ) ); 


—0.8541019662496845446 13760503093 
5.85410196624968454461376050310 | 


4, NullSpace. Consider 
> B := Matrix( 2, 2, [[666,667],[665,666]] ): A := Transpose(B) .B; 
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We make a floating-point version of A by 

> Af := Matrix( A, datatype=float[8] ); 

and then take the Nul1lSpace of both A and Af. The nullspace of A is correctly returned as 
the empty set — A is not singular (in fact, its determinant is 1). The nullspace of Af is correctly 


returned as 
—0.707637442412755612 
0.706575721416702662 ; 

The answers are different — quite different — even though the matrices differ only in datatype. 
The surprising thing is that both answers are correct: Maple is doing the right thing in each case. 
See [Cor93] for a detailed explanation, but note that Af times the vector in the reported nullspace 
is about 3.17 - 10~¥ times the 2-norm of Af. 

5. Approximate Jordan form (not!).! 
As noted previously, the Jordan form is discontinuous as a function of the entries of the matrix. This 
means that rounding errors may cause the computed Jordan form of a matrix with floating-point 
entries to be incorrect, and for this reason Maple refuses to compute the Jordan form of such a 
matrix. 
6. Conditioning of eigenvalues. 

To explore Maple’s facilities for the conditioning of the unsymmetric matrix eigenproblem, consider 
the matrix “gallery(3)” from MATLAB. 
> A := Matrix( [[-149,-50,-154], [537,180,546], [-27,-9,-25]] ): 

The Maple command EigenConditionNumbers computes estimates for the reciprocal 
condition numbers of each eigenvalue, and estimates for the reciprocal condition numbers of the 
computed eigenvectors as well. At this time, there are no built-in facilities for the computation of 
the sensitivity of arbitrary invariant subspaces. 


> E,V,rconds,rvecconds := EigenConditionNumbers( A, output= \ 
['values', ‘'vectors', 'conditionvalues','conditionvectors'] ): 
> seq( 1/rconds[i], i=1..3 ); 

417.6482708, 349.7497543, 117.2018824 


A separate computation using the definition of the condition number of an eigentriplet (y*, 4, x) 
(see Chapter 15) as 


C= IY loo lXllo0 
ly* + x| 


gives exact condition numbers (in the infinity norm) for the eigenvalues 1, 2, 3 as 399, 252, and 
147. We see that the estimates produced by EigenConditionNumbers are of the right order 
of magnitude. 


72.9 Canonical Forms 


There are several canonical forms in Maple: Jordan form, Smith form, Hermite form, and Frobenius 
form, to name a few. In this section, we talk only about the Smith form (defined in Chapter 6.5 and 
Chapter 23.2). 


1An out-of-date, humorous reference. 


72-16 Handbook of Linear Algebra 


Commands: 


1. SmithForm( B, output=['S','U','V']_ ) Smith form of B. 


Examples: 
The Smith form of 
ie —4y? 4y —1 ] 
—4y? Ay -1 0 
B= 
4y 1 4 (2y2—2) y 4 (y2-1)'y?-2y? +2 
AD. vata Py S27 42 —4(y?-1)'y 
is 
0 0 0 
01 0 0 
S= 
0 0 1/4 (2y?=1)° 0 
0 0 0 (1/64) (2 y? —1)° 


Maple also returns two unimodular (over the domain Q[y]) matrices u and v for which A = U.S.V. 


72.10 Structured Matrices 


Facts: 


1. Computer algebra systems are particularly useful for computations with structured matrices. 

2. User-defined structures may be programmed using index functions. See the help pages for details. 

3. Examples of built-in structures include symmetric, skew-symmetric, Hermitian, Vandermonde, and 
Circulant matrices. 


Examples: 


Generalized Companion Matrices. Maple can deal with several kinds of generalized companion matrices. 
A generalized companion matrix’ pencil of a polynomial p(x) is a pair of matrices Cy, C; such that 
det(xC,; — Co) = 0 precisely when p(x) = 0. Usually, in fact, det(xC; — Co) = p(x), though in 
some definitions proportionality is all that is needed. In the case C; = I, the identity matrix, we have 
Co = C(p(x)) is the companion matrix of p(x). MATLAB’s roots function computes roots of polynomials 
by first computing the eigenvalues of the companion matrix, a venerable procedure only recently proved 
stable. 

The generalizations allow direct use of alternative polynomial bases, such as the Chebyshev polynomials, 
Lagrange polynomials, Bernstein (Bézier) polynomials, and many more. Further, the generalizations allow 
the construction of generalized companion matrix pencils for matrix polynomials, allowing one to easily 
solve nonlinear eigenvalue problems. 

We give three examples below. 

If p := 3+ 2x4 x’, then CompanionMatrix( p, x ) produces “the” (standard) companion 
matrix (also called Frobenius form companion matrix): 


ae 


Sometimes known as “colleague” or “comrade” matrices, an unfortunate terminology that inhibits keyword search. 
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and it is easy to see that det(tl — C) = p(t). If instead 
p := Be (x) + 2B} (x) + 3B3(x) + 4B3 (x) 
where Bi (x) = are — x)"-*(x + 1)* is the kth Bernstein (Bézier) polynomial of degree n on the 


interval —1 < x < 1,thenCompanionMatrix( p, x _ ) produces the pencil (note that this is not in 
Frobenius form) 


=8)2:. te =a 
Gos | 2 Sa Ss 
0 1/2 = 

3 

3/2 0 —4 

c= |W/2 1/2 -8 
0 1/2 Zz 

3 


(from a formula by Jonsson & Vavasis [JV05] and independently by J. Winkler [Win04]), and we have 
p(x) = det(xC, — Co). Note that the program does not change the basis of the polynomial p(x) of 
Equation (72.9) to the monomial basis (it turns out that p(x) = 20 + 12x in the monomial basis, in this 
case: note that C; is singular). It is well-known that changing polynomial bases can be ill-conditioned, 
and this is why the routine avoids making the change. 

Next, if we choose nodes [—1, —1/3,1/3,1] and look at the degree 3 polynomial taking the values 
[1,—1,1,—1] on these four nodes, then CompanionMatrix( values, nodes ) gives Cy and C; 
where C; is the 5 x 5 identity matrix with the (5,5) entry replaced by 0, and 


- ibe 20% 16 ed 
6 13 0 & 4 
Get Oy Oh SB 0), 
Oe. -“Oe= OR Es SL 


16 «616 16 «16 


We have that det(tC, — Co) is of degree 3 (in spite of these being 5 x 5 matrices), and that this polynomial 
takes on the desired values +1 at the nodes. Therefore, the finite eigenvalues of this pencil are the roots of 
the given polynomial. See [CWO4] and [Cor04] for example, for more information. 

Finally, consider the nonlinear eigenvalue problem below: find the values of x such that the matrix C 
with Cj = To(x)/(Gi + j +1) + Ti(x)/(i+ j +2) + Th(x)/(i + j + 3) is singular. Here T(x) means the 
kth Chebyshev polynomial, T(x) = cos(k cos~!(x)). We issue the command 
> ( cO, Cl ) := CompanionMatrix( C, x ); 
from which we find 


2 

0 0 0 —2/15 i a 
0-0-0" =1712 1/24 
co=|0 0 0 z ia 
35 63 

100 -1/4 1/5 1/6 
Oe. 215 1/6 1/7 

00 1 —1/6 1/7 1/8 
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and 
1 0 0 0 0 0 
0 1 0 0 0 0 
0 0 1 0 0 0 
Cl= 
0 0 0 2/5 1/3 2/7 
0:0. 0 173 2/7 1/4 
0 0 0 2/7 1/4 2/9 


This uses a formula from [Goo61], extended to matrix polynomials. The six generalized eigenvalues of 
this pencil include, for example, one near to —0.6854 + 1.9097. Substituting this eigenvalue in for x in C 
yields a three-by-three matrix with ratio of smallest to largest singular values 03/0, © 1.7 - 107!°. This is 
effectively singular and, thus, we have found the solutions to the nonlinear eigenvalue problem. Again note 
that the generalized companion matrix is not in Frobenius standard form, and that this process works for 
a variety of bases, including the Lagrange basis. 
Circulant matrices and Vandermonde matrices. 
> A := Matrix( 3, 3, shape=Circulant[a,b,c] ); 


abe 
c a bi, 
boca 
> F3 := Matrix( 3,3,shape=Vandermonde[[1,exp(2*Pi*I/3),exp(4*Pi*I/3)]] 
)i 
1 1 1 


1 -1/2+4+1/2iv3 (-1/2+1/2iV3)’ |. 
1 -1/2—1/2iv3 (-1/2—1/2iV3)" 
It is easy to see that the F3 matrix diagonalizes the circulant matrix A. 


Toeplitz and Hankel matrices. These can be constructed by calling ToeplitzMatrix and Hankel- 
Matrix, or by direct use of the shape option of the Mat rix constructor. 


> T := ToeplitzMatrix( [a,b,c,d,e,f,g] ); 
> T := Matrix( 4,4,shape=Toeplitz[false,Vector(7,[a,b,c,d,e,f,g])] ); 
both yield a matrix that looks like 

dc b “] 

e dc b 

f 2 a.cel 

g fed 


though in the second case only 7 storage locations are used, whereas 16 are used in the first. This economy 
may be useful for larger matrices. The shape constructor for Toeplitz also takes a Boolean argument true, 
meaning symmetric. 
Both Hankel and Toeplitz matrices may be specified with an indexed symbol for the entries: 
> H := Matrix( 4, 4, shape=Hankel[a] ); yields 
4, a2 a3 ag 
az a3 ag a5 
43 a4 a5 a6 


44 45 ag a7 
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72.11 Functions of Matrices 


The exponential of the matrix A is computed in the MatrixExponential command of Maple by 
polynomial interpolation (see Chapter 11.1) of the exponential at each of the eigenvalues of A, including 
multiplicities. In an exact computation context, this method is not so “dubious” [Lab97]. This approach 
is also used by the general MatrixFunction command. 


Examples: 


> A := Matrix( 3, 3, [[-7,-4,-3],[10,6,4],[6,3,3]] ): 
> MatrixExponential( A ); 


6—7e! 3-4e! 2—3e! 
10e!—-6 -—3+6e! —2+4e! (72.1) 
6e!—6 —3+3e! -—2+3e¢e! 


Now a square root: > MatrixFunction( A, sqrt(x), x ): 


<6 =7/2 =5/2 
8 5 3 (72.2) 
6 3 3 


Another matrix square root example, for a matrix close to one that has no square root: 
> A := Matrix( 2, 2, [[epsilonA2, 1], [0, deltaA2] ] ): 
> S := MatrixFunction( A, sqrt(x), x ): 

> simplify( S ) assuming positive; 


1 


e+5 (72.3) 
0 6 


If € and 5 both approach zero, we see that the square root has an entry that approaches infinity. Calling 
MatrixFunction on the above matrix with e = 5 = 0 yields an error message, Matrix function 
xXxA(1/2) is not defined for this Matrix, which is correct. 

Now for the matrix logarithm. 


> Pascal := Matrix( 4, 4, (1,3)->binomial(j-1,i-1) ); 
sie Fale Cee | 
Oy 233 
(72.4) 
001 3 
000 1 
> MatrixFunction( Pascal, log(x), x ); 
0 1 0 0 
002 0 
(72.5) 
00 0 3 
00 0 0 


Now a function not covered in Chapter 11, instead of redoing the sine and cosine examples: 
> A := Matrix( 2, 2, [[-1/5, 1], [0, -1/5]] ): 
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> W := MatrixFunction( A, LambertW(-1,x), x ); 


LambertW (—1, —1/5) 
LambertW(—1, —1/5) 5 
Ws 1 + LambertW (—1, —1/5) (72.6) 


0 LambertW (—1, —1/5) 


> evalf( W ); 


—2.542641358 —8.241194055 
(72.7) 
0.0 —2.542641358 


That matrix satisfies W exp(W) = A, and is a primary matrix function. (See [CGH*96] for more details 
about the Lambert W function.) 
Now the matrix sign function (cf. Chapter 11.6). Consider 


> Pascal2 := Matrix( 4, 4, (1,j3)->(-1)A(i-1)*binomial(j-1,i-1) ); 
1 1 1 1 
0 -l -—2. -3 
(72.8) 
0 0 1 3 
0 0 0 -l1 
Then we compute the matrix sign function of this matrix by 
> S := MatrixFunction( Pascal2, csgn(z), z ): whichturnsoutto be the same matrix 


(Pascal2). 
Note: The complex “sign” function we use here is not the usual complex sign function for scalars 


signum(re’”) = 9, 
but rather (as desired for the definition of the matrix sign function) 
1 ifRe(z)>0 
csgn(z) = —1 ifRe(z) <0. 
signum(Im(z)) if Re(z) =0 


This has the side effect of making the function defined even when the input matrix has purely imaginary 
eigenvalues. The signum and csgn of 0 are both 0, by default, but can be specified differently if desired. 
Cautions: 


1. Further, it is not the sign function in MAPLE, which is a different function entirely: That function 
(sign) returns the sign of the leading coefficient of the polynomial input to sign. 

2. (In General) This general approach to computing matrix functions can be slow for large exact 
or symbolic matrices (because manipulation of symbolic representations of the eigenvalues using 
RootOf, typically encountered for n > 5, can be expensive), and on the other hand can be unstable 
for floating-point matrices, as is well known, especially those with nearly multiple eigenvalues. 
However, for small or for structured matrices this approach can be very useful and can give insight. 


72.12 Matrix Stability 


As defined in Chapter 19, a matrix is (negative) stable if all its eigenvalues are in the left half plane 
(in this section, “stable” means “negative stable”). In Maple, one may test this by direct computation of the 
eigenvalues (if the entries of the matrix are numeric) and this is likely faster and more accurate than any 
purely rational operation based test such as the Hurwitz criterion. If, however, the matrix contains symbolic 
entries, then one usually wishes to know for what values of the parameters the matrix is stable. We may 
obtain conditions on these parameters by using the Hurwitz command of the PolynomialTools 
package on the characteristic polynomial. 
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Examples: 


Negative of gallery (3) from MATLAB. 

> A := -Matrix( [[-149,-50,-154], [537,180,546], [-27,-9,-25]] ): 
> E := Matrix( [[130, -390, 0], [43, -129, 0], [133,-399,0]] ): 
> ACEH, pS Ala t Bs 


149 — 130¢t 50 + 390t 154 
—537—43t —180+129t —546 (72.9) 
27 — 133t 9+ 399t 25 


For which tf is that matrix stable? 
> p := CharacteristicPolynomial( AtE, lambda ); 
> PolynomialTools[Hurwitz]( p, lambda, 's', 'g' ); 
This command returns “FAIL,” meaning that it cannot tell whether p is stable or not; this is only to be 
expected as t has not yet been specified. However, according to the documentation, all coefficients of A 
returned in s must be positive, in order for p to be stable. The coefficients returned are 


Be iy (6+t)A (60 + 1733812 t + 49251217) a 
640" 154 4334537 41231282” (6+ (64 1221271) 


(72.10) 


and analysis (not given here) shows that these are all positive if and only if t > —6/1221271. 
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73.1 Introduction 


About Mathematica® 


Mathematica is a comprehensive software system for doing symbolic and numerical calculations, creating 
graphics and animations, writing programs, and preparing documents. 

The heart of Mathematica is its broad collection of tools for symbolic and exact mathematics, but 
numerical methods also form an essential part of the system. Mathematica is known for its high-quality 
graphics, and the system is also a powerful programming language, supporting both traditional procedural 
techniques and functional and rule-based programming. In addition, Mathematica is an environment for 
preparing high-quality documents. 

The first version of Mathematica was released in 1988. The current version, version 5, was released in 
2003. Mathematica now contains over 4000 commands and is one of the largest single application programs 
ever developed. Mathematica is a product of Wolfram Research, Inc. The founder, president, and CEO of 
Wolfram Research is Stephen Wolfram. 

Mathematica contains two main parts — the kernel and the front end. The kernel does the compu- 
tations. For example, the implementation of the Integrate command comprises about 500 pages of 
Mathematica code and 600 pages of C code. The front end is a user interface that takes care of the com- 
munication between the user and the kernel. In addition, there are packages that supplement the kernel; 
packages have to be loaded as needed. 

The most common type of user interface is based on interactive documents known as notebooks. 
Mathematica is often used like an advanced calculator for a moment’s need, in which case a notebook is 
simply an interface to write the commands and read the results. However, often a notebook grows to be a 
useful document that you will save or print or use in a presentation. 
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Notebooks consist of cells. Each cell is indicated with a cell bracket at the right side of the notebook. 
Cells are grouped in various ways so as to form a hierarchical structure. For example, each input is in a 
cell, each output is in a cell, and each input-output pair forms a higher-level cell. 


About this Chapter 


Within the limited space of this chapter, we cover the essentials of doing linear algebra calculations with 
Mathematica. The chapter covers many of the topics of the handbook done via Mathematica, and the 
reader should consult the relevant section for further information. 

Most commands are demonstrated in the Examples sections, but note that the examples are very simple. 
Indeed, the aim of these examples is only to show how the commands are used and what kind of result we 
get in simple cases. We do not demonstrate the full power and every feature of the commands. 

Commands relating to packages are frequently mentioned in this chapter, but they are often not fully 
explained or demonstrated. Mathematica has advanced technology for sparse matrices, but we only briefly 
mention them in Section 73.3; for a detailed coverage, we refer to [WR03]. The basic principle is that 
all calculations with sparse matrices work as for usual matrices. [WR03] also considers performance and 
efficiency questions. 

The Appendix contains a short introduction to the use of Mathematica. There we also refer to some 
books, documents, and Help Browser material where you can find further information about linear algebra 
with Mathematica. 

This chapter was written with Mathematica 5.2. However, many users of Mathematica have earlier 
versions. To help these users, we have denoted by (Mma 5.0) and (Mma 5.1) the features of Mathematica 
that are new in versions 5.0 and 5.1, respectively. In the Appendix we list ways to do some calculations 
with earlier versions. 

As in Mathematica notebooks, in this chapter Mathematica commands and their arguments are in 
boldface while outputs are in a plain font. Mathematica normally shows the output below the input, but 
here, in order to save space, we mostly show the output next to the input. 

Matrices are traditionally denoted by capital letters like A or M. However, in Mathematica we have the 
general advice that all user-defined names should begin with lower-case letters so that they do not conflict 
with the built-in names, which always begin with an upper-case letter. We follow this advice and mainly 
use the lower-case letter m for matrices. In principle, we could use most upper-case letters, but note that 
the letters C, D, E, I, N, and O are reserved names in Mathematica and they cannot be used as user-defined 
names. 

Because the LinearAlgebra MatrixManipulation™ package appears quite frequently in this 
chapter, we abbreviate it to LAMM. 
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Commands: 


Vectors in Mathematica are lists. In Section 73.3, we will see that matrices are lists of lists, each sublist 
being a row of the matrix. Note that Mathematica does not distinguish between row and column vectors. 
Indeed, when we compute with vectors and matrices, in most cases it is evident for Mathematica how an 
expression has to be calculated. Only in some rare cases do we need to be careful and write an expression 
in such a way that Mathematica understands the expression in the correct way. One of these cases is the 
multiplication of a column by a row vector; this has to be done with an outer product (see Outer in item 4 
below). 


1. Vectors in Mathematica: 
*{a, b, c, ...}A vector with elements a,b, ¢,.... 


* MatrixForm[v] Display vector v in a column form. 
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Length[v] The number of elements of vector v. 


VectorQ[v] Test whether v is a vector. 


2. Generation of vectors: 


Range[n,] Create the vector (1, 2,...,mq). WithRange[ng, n4] we get the vector (ng, 
no + 1,...,nz)andwithRange[ng, nz, 4] the vector (ng,ng + d,ng + 2d,...,mq). 
Table[expr,{i, nz}] Create a vector by giving i the values 1, 2,...,m4 in expr. If the 
iteration specification is {i, no, ny}; then i gets the values ng, ng + 1,..., m4, and for 
{i, ng, ny, 4}, the values of i are between ng and nq in steps of d. For {nq}, simply ny 
copies of expr are taken. 


Array[f£, nz] Createthen, vector (£[1],...,£[nz]).WithArray[£, nz, ng] we 
get the nq vector (£[ng],...,£[ng + ny — 1)). 


3. Calculating with vectors: 


a v Multiply vector v with scalar a. 
u + v Add two vectors. 


u v_ Multiply the corresponding elements of two vectors and form a vector from the products 
(there is a space between u and v); for an inner product, write u.v. 


u/v_ Divide the corresponding elements of two vectors. 
vp Calculate the pth power of each element of a vector. 


a*v_ Generate a vector by calculating the powers of scalar a that are given in vector v. 


4. Products of vectors: 


u.v_ The inner product of two vectors of the same size. 
Outer[Times, u, v] The outer product (a matrix) of vectors u and v. 


Cross[u, v] The cross product of two vectors. 


5. Norms and sums of vectors: 


Norm[v] (Mma 5.0) The 2-norm (or Euclidean norm) of a vector. 
Norm[v, p] (Mma5.0) The p-norm ofa vector (p is a number in [1, 00) or 00). 
Total[v] (Mma5.0) The sum of the elements of a vector. 


Apply[Times, v] The product of the elements of a vector. 


6. Manipulation of vectors: 


v[[il] Take element i (output is the corresponding scalar). 

v{{il] = a Change the value of element i into scalar a (output is a). 

vi ({i, j, ---}1] Take elements i, j, ... (output is the corresponding vector). 
First[v], Last[v] Take the first/last element (output is the corresponding scalar). 
Rest [v], Most[v] Drop the first/last element (output is the corresponding vector). 
Take[v, n], Take[v, -n], Take[v, {nz, ng}] Take the first n elements / the last 
n elements / elements nz, ...,ng (output is the corresponding vector). 


Drop[v, n], Drop[v, -n], Drop[v, {nz, ng}] Drop the first n elements / the last 
n elements / elements nz, ...,ng (output is the corresponding vector). 


Prepend[v, a], Append[v, a] Insertelementa atthe beginning/end ofa vector (output 
is the corresponding vector). 


Join[u, v, ...-] Join the given vectors into one vector (output is the corresponding 
vector). 


73-4 Handbook of Linear Algebra 


7. Inthe LinearAlgebra* Orthogonalization~ package: 


* GramSchmidt[{u, v, ...}] Generate an orthonormal set from the given vectors. 
* Projection[u, v] Calculate the orthogonal projection of u onto v. 


8. In the Geometry Rotations” package: rotations of vectors. 


Examples: 


1. Vectors in Mathematica. 


v= {4, 2, 3} {4, 2, 3} 


ws 


MatrixForm[v] 2 


Ww 


Length[v] 3 
VectorQ[v] True 


2. Generation of vectors. Range is nice for forming lists of integers or reals: 


Range[10] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} 
Range[0, 10, 2] {0, 2, 4, 6, 8, 10} 
Range[1.5, 2, 0.1] {1.5, 1.6, 1.7, 1.8, 1.9, 2.} 


Table is one of the most useful commands in Mathematica: 


Table[Random[], {3}] {0.454447, 0.705133, 0.226419} 
Table[Random[Integer, {1, 6}], {5}1] {2, 3, 6, 3, 4} 
Table[x*i, {i, 5}]  {x, x?, x3, x4, x°?} 
Table[x[il, {i, 5}] {x[1], x[2], x[3], x[4], x{[5]} 
Table[x,, {i, 5}] {x1, xo, x3, x4, x5} 

Array is nice for forming lists of indexed variables: 


Array[x, 5] {x[1], x[2], x[3], x[4], x{[5]} 
Array[x, 5, 0] {x[0], x[1], x[2], x[3], x[4]} 


3. Calculating with vectors. The arithmetic operations of multiplying a vector with a scalar and adding 
two vectors work in Mathematica as expected. However, note that Mathematica also does other 
types of arithmetic operations with vectors — multiplication, division, and powers. All arithmetic 
operations are done in an element-by-element way. For example, u vandu/v form a vector from 
the products or quotients of the corresponding elements of u and v. This is a useful property in 
many calculations, but remember to use u.v (which is the same as Dot [u, vw]) for an inner 
product. 


= {a, b, ch; v = {4, 2, 3}; 
{10 v, u + v, u v} 
{{40, 20, 30}, {4+ a, 2+b, 3 oe) {4 a, 2b, 3 c}} 


{1/v, u/v, wa} {{d. 7 3h, {3.3 ' sh, {a?, b*, <7} 


Functions of vectors are also calculated elementwise: 


Logiv] {Log[4], Log[2], Log[3]} 
4. Products of vectors. With u and was in Example 3, we calculate an inner product, an outer product, 


and a cross product: 


veu 4a+2b+36C 
Outer[Times, v, u] 
Ata: a, 4 by Ach; {2 a, 2 by. 2-e}, 43a, 3 5b; 3 et} 
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4a 4b 4c 
MatrixForm[%] 2a 2b 2c 
3a 3b 3c 


Cross[v, u] {-3b + 2c, 3a — 4c, —2a + 4b} 
5. Norms and sums of vectors. The default vector norm is the 2-norm: 


Norm[u] /Abs[a]2 + Abs[b]2 + Abs[c]2 


Norm[u, 2] \Abs[a]* + Abs[b]?2 + Abs[c]? 
Norm[u, 1] Abs[a] + Abs[b] + Abs[c] 
Norm[u, oo] Max[Abs[a], Abs[b], Abs[c]] 
Totalfu] at+bte 

Apply[Times, u] abc 


Applications: 


1. (Plotting of vectors) A package [WR99, p. 133] defines the graphic primitive Arrow which can 
be used to plot vectors. As an example, we compute the orthogonal projection of a vector onto 
another vector and show the three vectors (for graphics primitives like Arrow, Line, and Text, 
see [Rus04, pp. 132—146]): 


<< LinearAlgebra’ Orthogonalization~™ 
<< Graphics Arrow’ 
u = {5, 1}; v = {2, 4}; 
pr = Projection[v, u] {33, 15} 
Show[Graphics[{Arrow[{0, 0}, u], Arrow[{0, 0}, vl, 
Arrow[{0, 0}, pr],Line[{v, pr}], Text[”u”, u, {-1, 0}], 
Text(["v", v, {-1, —0.4}]}], Axes — True, 
AspectRatio — Automatic, PlotRange — All]; 


+- NW fF 


73.3 Basics of Matrices 


Commands: 


Matrices in Mathematica are lists of lists, each sublist being a row of the matrix. Although the traditional 
symbols for matrices are capital letters like A or M, we use the lower-case letter m for most matrices 
because, in Mathematica, all user-defined names should begin with a lower-case letter. 

Here we only briefy mention sparse matrices; a detailed exposition can be found in [WR03]; this 
document is the item Built-in Functions > Advanced Documentation > Linear Algebra in the Help Browser 
of Mathematica. 
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Matrices in Mathematica: 


*{{a, b, ...}, fe, d, ...}, ~.-} A matrix with rows (a,b,...), (¢,d,...),.... 
* MatrixForm[m] Display matrix min a two-dimensional form. 

* Length[m] The number of rows of matrix m. 

* Dimensions[m] The number of rows and columns of matrix m. 

* MatrixQ[m] Test whether mis a matrix. 


¢ The menu command Input > Create Table/Matrix/Palette generates an empty matrix into which 
the elements can be written. 


. Generation of matrices: 


* IdentityMatrix[n] Ann x n identity matrix. 
* DiagonalMatrix[{a, b, ...}] A diagonal matrix with diagonal elements a, b,.... 


* HilbertMatrix, HankelMatrix, ZeroMatrix, UpperDiagonalMatrix, 
LowerDiagonalMatrix, TridiagonalMatrix (in the LAMM package). 


* Table[expr, {i, mz}, {j, n4z}] Create a matrix by giving, in expr, i the values 1, 
2,...,mq and, for each i, j the values 1, ...,n4. Other forms of an iteration specification are 
{mz}, {i, mg, mz},and{i, mg, my, 4} (see item 2 in Section 73.2). 


Array[f£, {m,, n,}] Createanm4 x nq matrix with elements £[i, J]. 


Array[£, {mz, nz}, {mg, ng}] Createanmy x nq matrix using starting values mg 
and ng for the indices (the default values of mg and ng are 1). 


. Sparse matrices: 


* SparseArray[rules] (Mma 5.0) Create a vector or matrix by taking nonzero elements 
from rules and setting other elements to zero. 


* Normal[s] (Mma 5.0) Show a sparse vector or matrix s in the usual list form. 


* ArrayRules[m] (Mma 5.0) Show all nonzero elements of a vector or matrix as rules. 


. Arithmetic of matrices (see also Section 73.4): 


* am Multiply matrixm by scalar a. 
*m + n Add two matrices. 


*m n Multiply the corresponding elements of two matrices (giving the Hadamard product of the 
matrices; there is a space between m and n); for a proper matrix product, write m.n (see item 2 
in Section 73.4). 


* mp Calculate the pth power of each element of a matrix; for a proper matrix power, write 
MatrixPower[m, p] (see item 3 in Section 73.4). 


*m*-1 Calculate the reciprocal of each element of a matrix; for a proper matrix inverse, write 
Inverse [m] (see item 5 in Section 73.4). 
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5. Sums and trace of matrices: 


* Total[m] (Mma 5.0) The vector of column sums (i.e., the sums of the elements of each 
column). 


* Total [Transpose[m]] orMap[Total, m] (Mma5.0) The vector of row sums. 
* Total[Flatten[m]] orTotal[m, 2] (Mma5.0) The sum ofall elements. 
¢ Tr[m] Trace. 
*Tr[m, List] List of diagonal elements. 
6. Plots of matrices: 
* ArrayPlot[m] (Mma 5.1) Plot a matrix by showing zero values as white squares, the maxi- 
mum absolute value as black squares, and values between as gray squares. 
* ArrayPlot[m, ColorRules — {0 — White, _— Black}] (Mma5.1) Showall 


nonzero values as black squares (also Mat rixPlot [m] (Mma 5.0) from the LAMM package). 


Examples: 


1. Matrices in Mathematica. Matrices are lists of rows: 
m= {{5, 2, 3}, {4, 0, 2}} {{5, 2, 3}, {4, 0, 2}} 
Bo ade 3 


4 0 2 
The number of rows and the size of the matrix are 


MatrixForm [m] 


Length[m] 2 
Dimensions[m] {2, 3} 


With the menu command Input > Create Table/Matrix/Palette, we get a matrix with empty slots 
that can then be filled with the help of the Tab key: 


(3 3) 


With the TableSpacing option of MatrixForm we can control the space between rows and 
columns (the default value of the option is 1): 


5 2:3 


40 2 
Note that Mathematica will not do any calculations with a matrix that is in a matrix form! For 
example, writing 


MatrixForm[m, TableSpacing —> {.5, .5}] 


5.23 


4 0 2 
defines m to be the matrix form of the given matrix and Mathematica does not do any calculations 
with this m. Write instead 


m= {{5, 2, 3}, {4, 0, 2}} // MatrixForm 


pb 2°-3 

4 0 2 
or(m = {{5, 2, 3}, {4, 0, 2}$) // MatrixFormor do as we did above, define the 
matrix withm = {{5, 2, 3}, {4, 0, 2}} and then display the matrix with Matrix- 
Form[m]. 


MatrixForm[m = {{5, 2, 3}, {4, 0, 2}}] 
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2. Generation of matrices. 
IdentityMatrix[3] {{1, 0, oO}, {0, 1, Oo}, {0, 0, 1}} 
DiagonalMatrix[{3, 1, 2}] {{3, 0, O}, {0, 1, 0}, {0, 0, 2}} 
Table[Random[], {2}, {2}] 
{{0.544884, 0.397472}, {0.308083, 0.191191}} 
Table[1/(i + j — 1), {i, 3}, {3, 3}] 


fark. “ak die Ts. ek dee. Ey ll 
1 O1 3 2' 3° Af 3° A’ 5 


If, Which, and Switch are often useful in connection with Tab1e. In the following examples, 
we form the same tridiagonal matrix in three ways (we show the result only for the first example): 
Table[If[i == j, 13, Iffi == j — 1, 11, 
If{i == j + 1, 12, 01]1, {i, 4}, {5, 4}] // MatrixForm 
13 11 #0 0 
12 13 11 #0 
0 12) 13 21 
0 0 12° 13 
Table[Which[i == j, 13, i == j -— 1, 11, i==j+ 1, 12, True, 
01, {i, 4}, {i, 4}1]; 
Table[Switch[i — j, -1, 11, 0, 13, 1, 12,-_, 0], 
{i, 4}, {5, 4}1; 
Here are matrices with general elements: 
Table[f[i, jl, {i, 2}, {i, 2}] 
(ei Ay tlt. QU Ele Uy £12. 21h 
Table[f; 5, {i, 2}, {3, 2}] {{f1,1- f1,2}, {£2,1, f2,2}} 
Array[f, {2, 2}] {{f[1, 1], f[1, 2]}, {£12, 11, £12, 2]}} 
3. Sparse matrices. With SparseArray we can input a matrix by specifying only the nonzero 


elements. With Normal we can show a sparse matrix as a usual matrix. One way to specify the 
rules is to write them separately: 


SparseArray[{{1, 1} — 3, {2, 3} > 2, {3, 2} > 4}] 
SparseArray[<3>, {3, 3}] 
% // Normal {{3, 0, O}, {0, 0, 2}, {0, 4, O}} 
Another way is to gather the left- and right-hand sides of the rules together into lists: 
SparseArray[{{1, 1}, {2, 3}, {3, 2}} > {3, 2, 4}]1; 
Still another way is to give the rules in the form of index patterns. As an example, we generate the 
same tridiagonal matrix as we generated in Example 2 with Table: 
SparseArray[{{i_,i_}— 13, {i,j }/;i-—j==-1-> 11, 

4, Arithmetic of matrices. The arithmetic operations of multiplying a matrix with a scalar and 
adding two matrices work in Mathematica as expected. However, note that Mathematica also 
does other types of arithmetic operations with matrices, especially multiplication and powers. 
All arithmetic operations are done in an element-by-element way. For example, m n forms a 
matrix from the products of the corresponding elements of m and n. This is a useful property 
in some calculations, but remember to use m.n (which is the same as Dot [m, n]) for an inner 


product,MatrixPower[m, p] fora matrix power, and Inverse [m] fora matrix inverse (see 
Section 73.4). 


m= {{2, 5}, {3, 1}}; 
n= {{a, b}, {c, a}}; 
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3n {{3 a, 3 b}, {3 c, 3 a}} 

m+n {{2 +a, 5+ b}, {3 + c, 1 + dh} 
mn {{2 a, 5 b}, {3 c, a}} 

mo. flee yey tem aey} 

aa {1a B} 1B af} 


5. Sums and trace of matrices. With mas in Example 4, here are the column sums, row sums, the sum 
of all elements, and trace: 


{Total[m], Total[Transpose[m]], Total[Flatten[m]], Tr[m]} 
{{5, 6}, {7, 4}, 11, 3} 

6. Plots of matrices. 
ArrayPlot[{{5, 2, 3}, {4, 0, 2}}]; 


m = SparseArray [{{i-_,j-} /; Mod[i - j, 10] ==0— 1}, {30, 30}]; 
ArrayPlot [m] ; 


73.4 Matrix Algebra 


Commands: 
1. Transpose: 
* Transpose[m] Transpose. 
* ConjugateTranspose[m] (Mma 5.1) Conjugate transpose. 
2. Product: 
*m.n Product of two matrices. 
*m.v_ Product ofa matrix and a (column) vector. 
* v.m_ Product of a (row) vector and a matrix. 
3. Power and matrix exponential: 
* MatrixPower[m, p] pth power ofa square matrix. 


* MatrixExp[m] Matrix exponentiale™= 5° , zm! of a square matrix. 
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4, Determinant and minors: 

* Det [m] Determinant of a square matrix. 

* Minors[m] Minors ofa square matrix. 

* Minors[m, k] kth minors. 
5. Inverse and pseudo-inverse: 

* Inverse[m] Inverse of a square matrix. 

* PseudoInverse[m] Pseudo-inverse (of a possibly rectangular matrix). 
6. Norms and condition numbers: 

* Norm[m] (Mma 5.0) The 2-norm of a numerical matrix. 


* Norm[m, p] (Mma5.0) The p-norm ofa matrix (p is a number in [1, 00) or oo or Frobe- 
nius). 


* InverseMatrixNorm[m, p] The p-norm of the inverse (in the LAMM package). 
* MatrixConditionNumber[m, p] The p-norm matrix condition number (in the LAMM 
package). 
7. Nullspace, rank, and row reduction: 
* NullSpace[m] Basis vectors of the null space (kernel). 
* MatrixRank[m] (Mma5.0) Rank. 


* RowReduce[m] Do Gauss—Jordan elimination to produce the reduced row echelon form. 


Examples: 
1. Transpose. 
m= {{2+ 31, 1-21}, {3 — 41, 4 + I}}; 
2+ 33 14 - 21 
3 —- 41 44 1 
Do Bac Bree ‘" 


MatrixForm[m] ( 


MatrixForm[Transpose[m] ] : : 
1-21 4+ i 


: ; 2-31 34+ 4i 
MatrixForm[ConjugateTranspose[m] ] : ; 
14+ 21 4-1 
2. Product. Consider the following matrices and vectors: 
m= {{1, 2, 3}, {4, 5, 6}}; n= {{p, aj, {ry Ss}, {ty u}}e 
v = {1, 2}; w= {1, 2, 3}; 


Map[MatrixForm, {m, n, v, wh] 


pq 1 
qe 2). 3 1 

pel ee 382]. 4 an ee 
4 5 6 2 

tou 3 


Here are some products: 
mon {{p + 2r + 3t, q+ 2s + 3u}, {4p + 5r + 6t, 4q + 58 + 6u}} 


n.v {p+ 2g, r+ 2s, t + 2u} 
wen {p 2r 3t, q+ 2s + 3u} 


3. Power and matrix exponential. Powers can be calculated with Mat rixPower or by using the dot 
repeatedly: 
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k = {{2, 3}, {-2, 1}}s 

MatrixPower[k, 4] {{-—50, —63}, {42, -29}} 

k.k.k.k {{-50, —63}, {42, —29}} 

MatrixExp[k // N] {{—2.6658, 3.79592}, {-2.53061, —3.93111}} 
4, Determinant and minors. First, we consider a nonsingular matrix: 


m= {{6, 2, 3}, {2, 7, 53, {4, 2, 5}}3 
Det[m] 98 
Minors[m] {{38, 24, —11}, {4, 18, 4}, {-24, —10, 25}} 
Here is a singular matrix: 
mee 147 OT 25 Ths. fae Sa) Op The, {8x7 17 Be Shy 157 Ge. 47-5) 9 
Det[n] 0 


5. Inverse and pseudo-inverse. First, we consider the nonsingular matrix m from Example 4: 


ge 25 2 im 5 9 (12 12> 3, 79 
are inverse[m] { { 33, Z9 aah-{ aoa +3}, { a9 is: 48}} 
m.i—IdentityMatrix[3] {{0, 0, O}, {0, 0, O}, {0, 0, O}} 


Then, we consider the singular matrix n. The inverse does not exist, but the Moore—Penrose 
pseudo-inverse does exist: 


Inverse [n] 
Inverse::Sing : 
Matrix {t75 7 260 IN Loy yO Ty A857. 85-3 h9, Ab Ody. SEL. as 
singular. More... 
Inverse[{{7, 7, 2, 7}, {2, 5, 0, 7}, {8, 7, 8, 3}, 
{5, 6, 4, 5}}] 
ps = PseudoInverse[n // N] 
{{0.324527, —0.235646, —0.0236215, —0.129634}, 
{—0.054885, 0.0700676, 0.020876, 0.0454718}, 
{—0.256434, 0.102381, 0.121511, 0.111946}, 
{—0.0535186, 0.136327, —0.031972, 0.0521774}} 
Next, we check the four characterizing properties of a pseudo-inverse: 
{n.ps.n — n, ps.n.ps — ps, ConjugateTranspose[n.ps] — n.ps, 
ConjugateTranspose[ps.n] — ps.n} // Chop 


(IAG, 0%. 04 Oh, HOG Og Oe OS P03 Oy “Oe Ohy 105. 0n 508 OTS 
(POs. 09- Oe- Oh, 405. 07> 05 Who FO Gp O50). (07 205 02508), 
LODZ Abt WO) ps4 OG. Oy: 0. OV AO, Oe. 02 0h 405 OF Oy Oh, 
£60 ey. Oy)? 405 0, 05. O ke: 05 0 0 Oh oF Oe 0 Oy ONY 


For the option Tolerance of PseudoInverse, see SingularValueList in section 73.7. 
6. Norms and condition numbers. The default matrix norm is the 2-norm (the largest singular value): 


Norm[{{3, 1}, {4, 2}}] 15 + /221 
MatrixForm[m = {{p, aq}, {r, s}}] (: 


Norm[m, Frobenius] Abs [p1? + Abs[q]? + Abs[r]? + abs[s]? 
Norm[m, 1] Max[Abs[p] + Abs[r], Abs[q] + Abs[s]] 
Norm[m, oo] Max[Abs[p] + Abs[q], Abs[r] + Abs[s]] 
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Nullspace, rank, and row reduction. The null space (or kernel) of a matrix M consists of vectors 
v for which Mv = 0. NullSpace gives a list of basis vectors for the null space. The matrix 
multiplied with any linear combination of these basis vectors gives a zero vector. For a nonsingular 
matrix M, the null space is empty (Mathematica gives the empty list {}), that is, there is no 
nonzero vector v such that Mv = 0. The null space of the following matrix contains two basis 
vectors: 


p= {{7, 4, 8, 0}, {7, 4, 8, 0}, {1, 8, 3, 3hh; 
ns = NullSpace[p] {{12, -21, 0, 52}, {-4, -1, 4, O}} 
The matrix p multiplied with any linear combination of the basis vectors gives a null vector: 
p.(a ns[[1]] + b ns[[2]]) // Simplify {0, 0, 0} 
Matrix p is rank deficient: 
MatrixRank[p] 2 
The number of basis vectors of the null space plus the rank is the number of columns: 
Length[NullSpace[p]] + MatrixRank[p] 4 
The reduced row echelon form of a nonsingular square matrix is the identity matrix. Here is the 
reduced row echelon form of p: 


3 

I Oo a SS 

13 

RowReduce[p] // MatrixForm 01 o a 
4 52 

0 0 0 0 


The number of nonzero rows in the reduced row echelon form is the rank. RowReduce is also 
considered in item 6 of Section 73.9. 


Applications: 


1. 


(Prediction of a Markov chain) We classify days to be either wet or dry and assume that weather 
follows a Markov chain having the states wet and dry [Rus04, pp. 730—735]. From the weather 
statistics of Snoqualmie Falls in Western Washington, we have estimated the transition proba- 


e 0.834 0.166 
bility matrix to be P = . This means that if today is a wet day, then tomor- 
0.398 0.602 


row will be a wet day with probability 0.834 and a dry day with probability 0.166, and if today 
is a dry day, tomorrow will be a wet day with probability 0.398 and a dry day with probabil- 
ity 0.602. (See Application 1 of Section 4.3, or Chapter 54 for more information on Markov 
chains.) 

If j4(n) is a row vector containing the probabilities of the states for day n, then y(n) = 4(0)P”. 
Assume that today is Monday and we have a dry day, then 11(0) = (0, 1). We predict the weather for 
Friday: 

p = {{0.834, 0.166}, {0.398, 0.602}}; 
{0, 1}.MatrixPower[p, 4] {0.680173, 0.319827} 


Thus, Friday is wet with probability 0.68 and dry with probability 0.32. 


. (Permanent) (See Chapter 31 for the definition and more information about permanents.) A 


permanent of a square matrix M with n rows can be computed as the coefficient of the product 
X1X2-+-x, in the product of the components of the vector Mv, where v = (x1,...,X,).S0,a permanent 
can be calculated with the following program: 
permanent [m_]:=With[{v = Array[x, Length[m]]}, 
Coefficient [Apply[Times, m.v], Apply[Times, v]]] 
m= {{6, 2, 3}, {2, 7, 5}, {4, 2, 5}}3 
permanent[m] 426 
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73.5 Manipulation of Matrices 


Commands: 


1. 


Taking and resetting elements: 

*m[[i, j]] Take element (i, j) (output is the corresponding scalar). 

*m[[i, j]] = a Change the value of element (i, j) into scalar a (output is a). 

Taking, dropping, and inserting rows: 

* m[[i]] Take row i (output is the corresponding vector). 

*m[[{iq, ig, --.}1] Take the given rows (output is the corresponding matrix). 

* First[m], Last[m] Take the first/last row (output is the corresponding vector). 

* Rest [m], Most [m] Drop the first/last row (output is the corresponding matrix). 

* Take[m, n], Take[m, -n], Take[m, {n,, ng}] Take the first n rows / the last n 
rows / rows Ny, ...,;Ng (output is the corresponding matrix). 

* Drop[m, n], Drop[m, -n], Drop[m, {ny, no} Drop the first n rows / the last n 
rows / rows Ny, ...,;Ng (output is the corresponding matrix). 


* Prepend[m, r], Append[m, r] Insert row x at the beginning/end of a matrix (output 
is the corresponding matrix). 


* Insert[m, r, i] Insert rowr between rows i — 1 and i (so, x becomes row i) (output 
is the corresponding matrix). 


. Taking and dropping columns: 


*m[[Al1, j]] Take column j (output is the corresponding vector). 
*m[[All, {j4, jg, ---}1]] Takethe given columns (output is the corresponding matrix). 


* Take[m, All, n], Drop[m, {}, n] Take/drop the first n columns (output is the cor- 
responding matrix). 


Taking submatrices: 
*mU[{iq, ig, .--}, {54+ igs ---}1] Take the elements having the given indices. 


*Take[m, {i,, ig}, {54,/ 3g}1] Take the elements having row indices between i, 
and ig and column indices between j4 and jo. 


Combining and extending matrices: 
* Join[m,, mg, ..-] Put matricesm, one below the other. 


* MapThread[Join, {m,, mg, ...}] Put matricesm, side by side; another way: 
Transpose [Join [Transpose [mz], Transpose[mg], ...]]. 


* PadRight[m, {n,, ng}] Extendmwith zeros to anny x mg matrix. 

* PadRight[m, {n,, ng}, a] Extendmwith replicates of matrixatoanny x ng matrix. 
In the LAMM package: 

* Taking parts of matrices: TakeRows, TakeColumns, TakeMatrix, SubMatrix. 

* Combining matrices: AppendColumns, AppendRows, BlockMatrix. 

Permuting rows and columns: 

*m([{iq, ---, in}]] Permute the rows into the given order. 

* RotateLeft [m], RotateRight[m] Move the first/last row to be the last/first row. 


* RotateLeft[m, {0, 1}], RotateRight[m, {0, 1}] Move the first/last column to 
be the last/first column. 
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Flattening and partitioning: 
* Flatten[m] Flatten out a matrix into a vector by concatenating the rows. 


* Partition[v, k] Partition a vector into a matrix having k columns and as many rows as 
become complete. 


73.6 Eigenvalues 


Commands: 


1; 


Eigenvalues and eigenvectors: 


* CharacteristicPolynomial[m, x] Characteristic polynomial ofa square matrix [note 
that Mathematica defines the characteristic polynomial of a matrix M to be det(M — x1) while 
the handbook uses the definition det(xI — M)]. 


* Eigenvalues[m] Eigenvalues of a square matrix, in order of decreasing absolute value 
(repeated eigenvalues appear with their appropriate multiplicity). 


* Eigenvectors[m] Eigenvectors of a square matrix, in order of decreasing absolute value of 
their eigenvalues (eigenvectors are not normalized). 


* Eigensystem[m] Eigenvalues and eigenvectors of a square matrix. 

Options: 

* Eigenvalues, Eigenvectors, and Eigensystem have the options Cubics (Mma 5.0) 
and Quartics (Mma 5.0) with default value False; if you want explicit radicals for all cubics 
and quartics, set Cubics — True and Quartics — True. 

Largest eigenvalues and generalized eigenvalues: 

* Eigenvalues[m, k] (Mma5.0) Calculate the k largest eigenvalues; similarly for Eigen- 
vectors and Eigensystem. 


* Eigenvalues[{m, a}] (Mma5.0) Calculate the generalized eigenvalues of matrix m with 
respect to matrix a; similarly for Eigenvectors and Eigensystem. 


Examples: 


Ll, 


Eigenvalues. First, we calculate the characteristic polynomial of m in two ways: 


m= {{6, 2, 3}, {2, 7, 5}, {4, 2, 5}}; 
chp = CharacteristicPolynomial[m, x] 98 — 81 x 4+ 18 x2 — x3 
Det[m — xIdentityMatrix[3]] 98 — 81 x + 18 x* — x3 


Then, we calculate the eigenvalues in two ways: 


lam = Eigenvalues[m] {8 + J/15, 8 — V15, 2} 
Solve[chp == 0] {{x > 2},{x > 8 — V15}, {x > 8 + vV15}} 


Here are the eigenvectors: 


vec = Eigenvectors [m] 


hee -3 + ¥(8 + V15), ah, ‘ee -2 + £(8 — 15), uh, 


{-5, -14, 16}} 


An eigenvalue A of a matrix M and the corresponding eigenvector x should satisfy Mx = Ax. We 
check this for the first eigenvalue and then for all the eigenvalues: 
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m.vec[[1]] — lam[[1]] vec[[1]] // Simplify {0, 0% oO} 
m.Transpose[vec] — Transpose[vec] .DiagonalMatrix[lam] // 
Simplify 


{{0, 0, Oo}, {0, 0, O}, {0, 0, OF} 


With Eigensystem we can compute both the eigenvalues and the eigenvectors. The result is a 
list where the first element is a list of eigenvalues and the second element a list of the corresponding 
eigenvectors: 
{lam, vec} = Eigensystem[m] 
{{g8 + /15, 8 — 15, 2}, 
{{1, -2 + 2(e + vis), a}. (2, -2 + 2e - VIB), 4}, 
{-5, —14, 16}}} 

2. Options. For more complex eigenvalues, we may get only representations of the values as some 
root objects, but with N we get numerical values and with the Cubics and Quartics options 
we may get explicit eigenvalues. Note that often we only need the decimal values of the eigenvalues 
(not their exact expressions), and then we can input the matrixm // N. 


n = {{7, 7, 2, 7}, {2, 5, 0, 7}, {8, 7, 8, 3}, {5, 6, 4, 5}}3 
Eigenvalues [m] 
{Root [-171 + 112 #1 — 25 #1? + #17 &, 1], 
Root[-171 + 112 #1 — 25 #17 + #1° &, 3], 
Root[-171 + 112 #1 — 25 #17 + #1° &, 2], 0} 
% // NM {19.7731, 2.61345 + 1.34834 i, 2.61345 — 1.34834 i, 0.} 
Eigenvalues[n // N] 
{19.7731, 2.61345 + 1.34834 i, 2.61345 — 1.34834 i, 
—2.85924 x 10716} 
Eigenvalues[n, Cubics — True] [[1]] 


1/3 1 
23 4 5 (29582 siya) + (F (20667 + 261/253)) 


#3 


Applications: 


1. (Positive and negative definite matrices) A symmetric (or Hermitian) matrix is positive [negative] 
definite if and only if all of the eigenvalues are positive [negative]. (See Section 8.4 for more 
information about positive definite matrices.) With this result we can easily test the definiteness of 
a matrix (note that Mathematica does not have a special command for this test). For example, the 
following symmetric matrix is positive definite: 


m = {{2, 0, 1}, {0, 4, 2}, {1, 2, 3}}; 
Eigenvalues[m] // N {5.66908, 2.47602, 0.854897} 


2. (A sufficient condition for a minimum or maximum) A sufficient condition for a local mini- 
mum [maximum] point of a function is that the Hessian is positive [negative] definite [BSS93, 
p. 134]. Calculate the stationary points of a function (for a program for classical optimization with 
constraints, see [Rus04, pp. 543—547]): 


f=x'34+x’°2y24+2xy-3x-y 

3x4¢x —-yt2xytx vy? 
grad = {D[f, x], DIf, yl} 

(8 3 Ao Ol a a Bey ee se 8 yt 
stat = Solve[grad == 0, {x, y}] // N 


{{y > 1.22959, x > -1.16294}, 

{y > 0.956992, x> 0.369407}, {y > —0.497601, x > 1.07442}, 
{y > —0.844491 — 2.24741 i, x > —0.140442 — 0.584262 i}, 
{y > -0.844491 + 2.24741 i, x > —0.140442 + 0.584262 i}} 
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Pick the real points (see Section 73.5, the second command in item 2 of Commands): 


stat2 = stat[[{1, 2, 3}]] 
{{y — 1.22959, x > —1,16294}, 
fy > 0.956992, x > 0.369407}, {y > —0.497601, x > 1.07442}} 


Calculate the Hessian and its eigenvalues at the real stationary points: 


hes = {{D[f£, x, x], DI[f, x, yl}, {DIf, y, xl, DI[£, y, yl}} 
{{ex+2y*,2+4xy}, (2+4xy, 2 x*}} 
Map[Eigenvalues[hes /. #] &, stat2] 

{{-5.61664, 4.36764}, {6.06166, —1.74063}, {6.94587, 2.30462}} 


Thus, the first two points are saddle points while the third point is a point of minimum. Note that 
the gradient and the Hessian can in Mathematica 5.1 also be calculated as follows: 


grad = D[f, {{x, y}, 1}] 
PES Bia aS pay ee ye 9 Se ee) 
hes = D[f, {{x, y}, 2h] 
{£6 see 2 Fy, 2 pod aay ADs et Sey, “2H 
3. (Eigenvector—eigenvalue decomposition) Let diag(A) be a diagonal matrix whose diagonal elements 
are the eigenvalues of a matrix M, and let P be a matrix whose columns are the eigenvectors 
of M. If P is nonsingular, we have the decomposition M = Pdiag(A)P~' or diagonalization 
PMP = diag(A). The former equality shows that M and diag(A) are similar matrices, so that, 
for example, they have the same determinant, trace, and rank. (See Section 4.3 for more information 
about diagonalization.) As an example of the eigenvector—eigenvalue decomposition, first compute 
the eigenvalues and eigenvectors: 
m= {{6, 2, 3}, {2, 7, 5}, {4, 2, 5}}3 
{lam, vec} = Eigensystem[m] 
{ep 4/15, 6 = (15, 2h, 
9 1 9 1 
{{1, -3 + 3 (8 + v5), 1}, {1, -3 + 3 (8 - VTS), 1}, 
{-5, —14, 16}}} 
Then, define the diagonal matrix of eigenvalues and the matrix whose columns are the eigenvectors: 


diag = DiagonalMatrix[lam] 
{{8 + V15, 0, 0}, {0, 8 — 715, O}, {0, 0, 2}} 


p = Transpose [vec] 
{{1, 1, —5}, { 244 (8+ VI5), -2 + 4 (8 — VI5), -14}, 


ce 1, 16}} 


The following computation verifies the decomposition: 


p.diag.Inverse[p] — m // Simplify 


(4050030). FO) Oy vOte POy “Oy sOFE 


73.7 Singular Values 


Commands: 
1. Singular values: 


* SingularValueList[m] (Mma 5.0) Nonzero singular values of a (possibly rectangular) 
numerical matrix with at least one entry that has a decimal point; the singular values are given in 
order of decreasing value (repeated singular values appear with their appropriate multiplicity). 
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2. Singular value decomposition: 


* SingularValueDecomposition[m] (Mma5.0) Singular value decomposition ofa (pos- 
sibly rectangular) numerical matrix with at least one entry that has a decimal point. For matrix 
M, the decomposition is M = UWV* where U and V are orthonormal and W is diago- 
nal with singular values as the diagonal elements (* means conjugate transpose). Output is 
{U, W, V}. 


3. An option: 


* The default value Automatic of the option Tolerance (Mma 5.0) keeps only singular values 
larger than 100 x 10~?, where p is Precision[m]. Tolerance — t keeps only singular 
values that are at least t times the largest singular value. Tolerance — 0 returns all singular 
values. 


4. Largest singular values and generalized singular values: 


* SingularValueList[m, k] (Mma5.0) Calculate the k largest singular values. 


* SingularValueList[{m, a}] (Mma5.0) Calculate the generalized singular values ofm 
with respect to a. 


Examples: 


1. Singular values. The singular values of a matrix M are the square roots of the eigenvalues of the 
matrix M*M: 


m= {{6, 2, 3}, {2, 7, 5}, {4, 2, 5}}; 
SingularValueList[m // N] {12.1048, 4.75064, 1.70418} 
Sqrt [Eigenvalues [ConjugateTranspose[m].m]] // N 
{12.1048, 4.75064, 1.70418} 


Only nonzero singular values are returned by SingularValueList: 


me=4{7, 7, 2, The (25 Se Oy Thy {8s Fz Sr Bh, [Sr 6e 4, Shh9 
SingularValueList[n // N] {21.5225, 7.09914, 1.84001} 


By giving Tolerance the value 0, we get all singular values: 


SingularValueList[n // N, Tolerance — 0] 
{21.5225, 7.09914, 1.84001, 5.46569 x 10717} 


2. Singular value decomposition. Here is an example of this decomposition: 


Map[MatrixForm, {u, w, v} = SingularValueDecomposition[m // N]] 
—0.516443 0.629389 —0.580652 


—0.672542 —0.717852 —0.179933], 
—0.53007 0.297588 0.79402 


12.1048 0. 0. 
0. 4.75064 0. : 

0. 0. 1.70418 
—0.542264 0.743264  -—0.391801 


—0.561826 —0.667491 —0.488678 
—0.624741 —0.0448685 0.779542 
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The following verifies the decomposition: 


m — u.w.ConjugateTranspose[v] // Chop 


{{0, 0, oO}, {0, 0, O}, {0, 0, OF} 


u.ConjugateTranspose[u] // Chop 


Tis. 0, OO dey Ole 40, Geo toy 


v.ConjugateTranspose[v] // Chop 


PETS Ore SON, Oe, Deg AON, CO 05° Mey 


Applications: 

1. (Condition numbers) Relative to a matrix norm || M||, the condition number of an invertible matrix 
M is ||M||||M~"||. (See Chapter 37 for more information about condition numbers.) A condition 
number is >1; a large condition number indicates sensitivity to round-off errors. The 2-norm 
condition number can be shown to be the maximum singular value divided by the minimum 
singular value: 

cond[m] := With[{s = SingularValueList[m // N, 
Tolerance — 0]}, First[s]/Last[s]] 


For example, consider the condition numbers of Hilbert matrices: 


h[n] := Table[1/(i + j — 1), {i, n}, {3, n}] 
Table[cond[h[il], {i, 1, 6}] 
{Tes 19.2815, 524.057, 15513.7, 476607., 1.49511 x 107} 


The condition numbers grow rapidly and they are large even for small Hilbert matrices. This 
indicates numerical difficulties associated with Hilbert matrices. Note that the LAMM package 
defines both HilbertMatrix and MatrixConditionNumber. 


73.8 Decompositions 


Commands: 
1. Decompositions into triangular matrices: 


* LUDecomposition[m] PLU decomposition of a square matrix. For matrix M, the decom- 
position is PM = LU, where L is unit lower triangular (with ones on the diagonal), U is upper 
triangular, and P is a permutation matrix. Output is {K, p,c}, where matrix K contains both L 
and U, p is a permutation vector, that is, a list specifying rows used for pivoting, and c is the L® 
condition number of M (however, for exact matrices with no decimal points, c is 1). 

* CholeskyDecomposition[m] (Mma 5.0) Cholesky decomposition of a Hermitian, posi- 
tive definite matrix. For matrix M, the decomposition is M = U*U, where U is upper triangular. 
Output is U. 


2. Orthogonal decompositions: 


* SingularValueDecomposition[m] (Mma5.0) Singular value decomposition ofa (pos- 
sibly rectangular) numerical matrix with at least one entry that has a decimal point. For matrix 
M, the decomposition is M = UWV* where U and V are orthonormal and W is diagonal with 
singular values as the diagonal elements (* means conjugate transpose). Output is {U, W, V}. See 
item 2 in Section 73.7. 

* QRDecomposition[m] QR decomposition of a numerical matrix. For matrix M, the de- 
composition is M = Q* R, where Q is orthonormal and R is upper triangular. Output is {Q, R}. 


3. Decompositions related to eigenvalue problems: 


* JordanDecomposition[m] Jordan decomposition of a square matrix. For matrix M, the 
decomposition is M = SJ S~!, where S isa similarity matrix and J is the Jordan canonical form 
of M. Output is {S, J}. 
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* SchurDecomposition[m] Schur decomposition ofa square, numerical matrix with at least 
one entry that has a decimal point. For matrix M, the decomposition is M = QT Q*, where Q 
is orthonormal and T is block upper triangular. Output is {Q, T}. 


* HessenbergDecomposition[m] (Mma5.1) Hessenberg decomposition of a square, nu- 
merical matrix with at least one entry that has a decimal point. For matrix M, the decomposition 
is M = P HP*, where P is an orthonormal and H a Hessenberg matrix. Output is { P, H}. 


* PolarDecomposition[m] Polar decomposition of a numerical matrix. For matrix M, the 
decomposition is M = US, where U is orthonormal and S is positive definite. Output is {U, S}. 
(In the LAMM package.) 


Examples: 


1. Decompositions into triangular matrices. As an example, we consider the PLU decomposition: 


m= {{2., 7, 5}, {6, 2, 3}, {4, 2, 5}}; 


Map[MatrixForm, {lu, perm, cond} = LUDecomposition[m] ] 
6. 0.333333 0.666667 2 
2. 6.33333 0.105263], ]1], 9.42857 
3. 4. 2.57895 3 


The matrix 1u contains both the L and the U matrix, perm is a permutation vector specifying 
rows used for pivoting, and cond is the L® condition number ofm. With a package we can extract 
the L and U matrices from the Lu matrix: 


<< LinearAlgebra* MatrixManipulation 
Map[MatrixForm, {1, u} = LUMatrices[lu]] 


3 0: O. Ore 2 3% 
0°. 3:3:3333% <L. O.], ]0 6.33333 4. 
0.666667 0.105263 1. 0 0 2.57895 


The following calculation verifies the PLU decomposition (see Section 73.5, the first command in 
item 7 of Commands): 


ml = m[[perm]] {{6, 2, 3}, {2., 7, 5}, {4, 2, 5}} 
ml — l.u {{0., 0., 0.}, {0., 0., O.}, {0., 0., 0.}} 


We can also verify the decomposition with the corresponding permutation matrix: 


permMatr = IdentityMatrix[3] [ [perm] ] 
Og: ee SOV Aly Oy OS 05 SOg AVF, 
m2 = permMatr.m {{6., 2., 3.}, {2., 7., 5.}, {4., 2., 5.}} 
m2 — 1l.u {{0., 0., 0.}, {0., 0., O.}, {0., 0., O.}} 
Once the PLU decomposition ofm is ready, we can solve a system of linear equations with coefficient 
matrix m. Define b, the constant vector, and use then LUBackSubstitution: 
b => {2, 4, 1}; 
LUBackSubstitution[{lu, perm, cond}, b] 
{0.826531, 0.530612, —0.673469} 
However, the PLU decomposition and the back substitution are automatically done with Lin- 
earSolve: 


LinearSolve[m, b] {0.826531, 0.530612, -0.673469} 
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73.9 Linear Systems 


Commands: 


1. Solving equations written explicitly: 


* lhs == rhs An equation with left-hand side 1hs and right-hand side rhs (after you have 
written the two equal signs ==, Mathematica replaces them with a corresponding single symbol). 
* eqns = {eqn4 , CGNg, «- . } A system of equations with the name eqns. 


* Solve[eqns] Solve the list of equations eqns for the symbols therein. 
* Solve[eqns, vars] Solve the list of equations eqns for the variables in list vars. 
* Reduce[eqns, vars] Give a full analysis of the solution. 
2. Solving equations defined by the coefficient matrix and the constant vector: 
* LinearSolve[m, v] Solve the linear systemm.vars == v. 
* Solve[m.vars == v] Solve the linear systemm.vars == v. 


* If it is possible that the system has more than one solution, it is better to use Solve, as 
LinearSolve only gives one solution (with no warning). 


3. Conversion between the two forms of linear equations: 


* eqns = Thread[m.vars == v] Generate explicit equations from a coefficient matrix 
m and a constant vector v by using the list of variables vars. 


*{m, v} = LinearEquationsToMatrices[eqns, vars] Give the coefficient matrix 
m and the constant vector v of given linear equations eqns containing variables vars (in the 
LAMM package). 


4. Solving several systems with the same coefficient matrix: 


* £ = LinearSolve[m] (Mma 5.0) Give a function £ for which £[v] solves the system 
m.x == Vv. 
* LUBackSubstitution[lu, v] Solve the systemm.x == v;herelu = LUDecom- 


position[m]. 


5. Using the inverse and pseudo-inverse: 


* Inverse[m] .v_ Give the solution of the equationsm.x == v (this method is not recom- 
mended). 
* PseudoInverse[m] .v_ Give the least-squares solution of the equationsm.x == v. 


6. Special methods: 


* RowReduce[a] Do Gauss—Jordan elimination to the augmented matrix a to produce the 
reduced row echelon form, where the last column is the solution of the system. 


* TridiagonalSolve[sub, main, super, rhs] Solve a tridiagonal system (in the 
LinearAlgebra  Tridiagonal~ package). 


7. Eliminating variables: 
* Solve[eqns, vars, elims] Solve eqns for vars, eliminating elims. 


* Eliminate[eqns, elims] Eliminate elims from eqns. 


Examples: 


1. Solving equations written explicitly. If the system does not have any symbols other than the variables 
for which the system has to be solved, we need not declare the variables in Solve: 
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sol = Solve[{2x+5y+4+2==2, 3x4+4y4+32 == 2, 
6x+y+ 22 == 5}] 
3 


47 5 
{ {x ~ Tg: Y > agra > -3 ht 
sol // nN {{x > 0.959184, y > 0.102041, z> -0.428571}} 


Here is an example where we declare the variables: 


eqns = {2x+5y+2z2==2,3x+4y4+32 == 2, 
6x+y+ 22 == 5}; 
vars = {x, yr zhi 


= 47 5 3 
sol = Solve[eqns, vars] {{x Mg Ge 2 +h} 


Solve gives the solution in the form of transformation rules. If we want a list of the values of the 


variables, we can write 
vars /. sol[[1]] {45 ; Bp: —3} 
To check the solution, write 
eqns /. sol {{True, True, True}} 
(Note that for an inexact solution the test may yield False for some equations due to round-off 
errors.) If a solution does not exist, we get an empty list: 


Solve[{x + 2 y == 2, 2x + 4y == 1}] {} 

If an infinite number of solutions exist, we get all of them (with a message): 
Solve[{x + 2 y == 2, 2x+4y == 4}] {{x > 2 — 2 y}} 
Solve::svars : Equations may not give solutions for all solve 


variables. More... 


This means that x has the value 2 — 2y, while y can be arbitrary. The message tells that the 
equations may not determine the values of all of the variables (this is, indeed, the case in our 
example because the solution does not give the value of y). 

The system may contain symbols: 


eqns2 = {2x +5 y==3, 3x+4d y == 2}; 
10: =. 3d 5 
Solve[eqns2, {x, y}] x > S152 at) Si 2 


Note that this is a generic solution that is valid for most values of the parameter d. To get a full 
analysis of the equations, use Reduce. The result is a logical statement where, for example, && 
means logical AND and || means logical OR: 


Reduce[eqns2, {x, y}] 
-15+2ad40 s& x== oPt39 ge yaad (3-2 x) 


2. Solving equations defined by the coefficient matrix and the constant vector. We continue with the 
same system that we considered in Example 1: 


= {{2, 5, 1}, {3, 4, 3}, {6, 1, 2}}; v = {2, 2, 5}; 

sol = LinearSolvel[m, v] 45, By: -3 

sol // N {0.959184, 0.102041, -0.428571} 
We can check the solution: 

m.sol — v_ {0, 0, 0} 
We can also use Solve if we form an equation with variables: 

Solve[m.{x, y, z} == v] { {x > ay: Yee egy es +}} 
If a solution does not exist, LinearSolve gives a message: 


LinearSolve[{{1, 2}, {2, 4}}, {2, 1}] 
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LinearSolve::nosol : Linear equation encountered which has no 
solution. More... 
LinearSolve[{{1, 2}, {2, 4}$}, {2, 1}] 


If an infinite number of solutions exist, we are given one of them (with no warning): 


LinearSolve[{{1, 2}, {2, 4}}, {2, 4}] {2, 0} 


The system may contain symbols: 
LinearSolve[{{2, 5}, {3, da}}, {3, 2}] 
5 


Shee. 3d 
=I 20d" =T5i4- 2° 
Conversion between the two forms of linear equations. Consider again the familiar system: 
eqns = {2x+5y+2z2==2, 3x+4y+32 == 2, 
6x+y+ 22 == 5}; 


The corresponding coefficient matrix and constant vector are 


<< LinearAlgebra’ MatrixManipulation~™ 
{m, v} = LinearEquationsToMatrices[eqns, {x, y, z}] 


{{{2, 5, 1}, {3, 4, 3}, {6, 1, 2}}, {2, 2, 5}} 


If we know mand v, we can obtain explicit equations as follows (here we use indexed variables): 


eqns = Thread[m.Table[x;, {i, 3}] == v] 
{2 ey ES ages KS == 2, 3 x, + 4 x9 + 3 x3 == 2, 
6 x1 + x2 + 2 x3 == 5} 


. Solving several systems with the same coefficient matrix. Suppose that we have one coefficient 


matrix m, but we would like to solve three systems by using the constant vectors v1, v2, and v3: 


m= {{2, 5, 1}, {3, 4, 3}, {6, 1, 2}}; 
vl = {2, 2, 5}; v2 = {1, 4, 3}; v3 = {4, 2, 6}; 


First, we create a function with LinearSolve: 
f = LinearSolve[m] LinearSolveFunction[{3, 3}, "<>"] 
Then, we give the three constant vectors as arguments: 
{f£Iv1], £[v2], £[v31} 
AT Br | 8s3 2. = FLO 68 26 _10 
49' 49' Tf! 49' 49" 7 f7 49' 49' a 


In this way we solve three systems, but the PLU decomposition is done only once, thus saving 
computing time. We could also use LUDecomposition: 


lu = LUDecomposition [nm] ; 
Map [LUBackSubstitution[lu, #] &, {v1, v2, v3}] 


47 5 oo 2 IRD) TO 68 26 _10 
49’ 49’ 7S! 49’ 49" 7 5! 49’ 49' 7 


. Using the inverse and pseudo-inverse. The inverse of the coefficient matrix can, in principle, be 


used to solve a linear system: 


m= {{2, 5, 1}, {3, 4, 3}, {6, 1, 2}}; v = {2, 2, 5}; 
Inverse [m] .v {43 ; a i -3 

However, this method should not be used, since inverting a matrix is a more demanding task than 
solving a linear system with Gaussian elimination. If the coefficient matrix is singular or nonsquare, 
we can, however, use a pseudo-inverse to calculate a least-squares solution. (See Sections 5.7, 5.8, 
and 5.9 for definitions and more information on the pseudo-inverse and least squares solution.) If 
the system is Mx = v, the least-squares solution minimizes the norm || Mx — v||2. As an example, 
consider a system with coefficient matix p and constant vector ¢: 
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p = {{7, 4, 8, 0}, {7, 4, 8, O}, {1, 8, 3, 3}}; c = {4, 1, 2}; 


The equations do not have any solutions in the usual sense, but a least-squares solution can be 
calculated: 


671 565 1907 201 
PEqusORHVer ae tElEe feces 3369’ 13476" t35} 


6. Special methods. RowReduce can be used to solve a system of equations. First we form the 
augmented matrix by appending the constant vector into the rows of the coefficient matrix: 
m= {{2, 5, 1}, {3, 4, 3}, {6, 1, 2}}; v = {2, 2, 5}; 
a = Transpose[Append[Transpose[m], v]] 
(195) ay ok, 435 ay 3, Dh, Fey od, Oy Bit 


Then, we do Gauss—Jordan elimination to produce the reduced row echelon form; the solution is 
the last column: 


7. Eliminating variables. The following equations contain the parameter d, which we would like to 


g = RowReduce [a] 


47 5 
{{2, 0, 0, #8h, {o, 1, 0, gh, fo, 0, 2, - 


sol = gifall, 411 {45, Z, -4} 


~J|Wo 


eliminate: 
eqns = {x y z==-d, x 2y+2== -d, 
-x + y + z == 2 ad}; vars = {x, y, z}; 
Solve[eqns, vars, {d}] { {x > 32, yo 52h} 
Solve::svars : Equations may not give solutions for all solve 


variables. More... 
Alternatively, we can use Eliminate. The result is then expressed as a logical expression: 


Eliminate[eqns, {d}] 5 x==3 2 && 5 y==6z2 


Applications: 


1. (Stationary Distribution of a Markov chain) The stationary distribution a (a row vector) of a 
Markov chain with transition probability matrix P is obtained by using the equations mP = a 
and >; = 1. One of the equations of the linear system 1 P = x can be dropped. We continue 
Application 1 of Section 73.4: 


p = {{0.834, 0.166}, {0.398, 0.602}}; st = {141 Toh? 
eqns = Append[Most [Thread[st.p == st]], Total[st] == 1] 
{0.834 2, + 0.398 mp == m4, 41 + TQ == 1} 

Solvefeqns] {{z, — 0.705674, my > 0.294326}} 


So, in the long run, about 71% of days are wet and 29% dry. 


73.10 Linear Programming 


Commands: 
1. Solving linear optimization problems written explicitly: 


* Minimize[{obj, cons}, vars] (Mma 5.0) Give the global minimum of obj subject 
to constraints cons with respect to variables vars. 


* Maximize[{obj, cons}, vars] (Mma5.0) Give the global maximum of obj subject 
to constraints cons with respect to variables vars. 
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2. Solving linear optimization problems defined by vectors and matrices: 

* LinearProgramming[c, a, b] Minimizec.xsubjecttoa.x > bandx > 0. 

* LinearProgramming[c, a, {{by, 81}, {bg, Sg}, «--}] Theithconstraintis 
a,-x < by,ay.x == by,oray.x > by according to whether s; is<0,=0,or> 0. 
Examples: 


1. Solving linear optimization problems written explicitly. 
obj =~ 2x+ 3 y; 
cons = {x +y>4, x+3y2>5, x*>0,y 
vars = {x, y}; 


IV 


0}; 


Minimize[{obj, cons}, vars] {¥, {x > 5, yo sh} 


Here the minimum value of the objective function is 17/2. Next, we solve an integer problem. The 
constraint vars € Integers or Element[vars, Integers] restricts the variables in 
vars to be integer-valued. 
Minimize[{obj, cons, vars € Integers}, vars] 
{9, {x > 3, y > 1}} 
2. Solving linear optimization problems defined by vectors and matrices. We solve the first problem 
in Example 1: 


¢ = {2, 3}; a= {{1, 1}, {1, 3}}7 b= {4, 5}; 


sol = LinearProgramming[c, a, b] Z, 5 
17 
c.sol a 


Applications: 


1. (A transportation problem) (See Chapter 50 for definitions and more information about linear 
programming.) Consider the following transportation problem (cf. [Rus04, pp. 524—526]). Sup- 
plies of plants 1 and 2, demands of cities 1, 2, and 3, and costs between the plants and the cities are 
as follows: 

supply = {40, 60}; demand = {35, 40, 25}; 

costs = {{4, 9, 6}, {5, 3, T7}}; 
For example, to transport one unit from plant 1 to city 3 costs $6. The problem is to minimize the 
total cost of transportation. Let x;,; be the amount transported from plant i to city j: 

vars = Table[x;4, {i, 2}, {j, 3}] 

{{x1,1/ ¥1,2- ¥1,3}, {*2,1/ *2,2) ¥2,3}} 

The objective function is 

obj = Flatten[costs] .Flatten[vars] 

4X,,1 + 9 *1,2 + 6 X1,3 + 5 %2,1 + 3 X2,2 + 7 X2,3 
The supply, demand, and nonnegativity constraints are as follows: 

sup = Thread[Map[Total, vars] < supply] 

44: tpg + X14, 3) S 407 eo 4 ke o> Fg 3 S 60} 

dem = Thread[Total[vars] > demand] 

SA Rl sate: 7 Age ee i ee eS i 

non = Thread[Flatten[vars] > 0] 

Tig. eC eer pe Oy Ra Oe eo at, On eo Oe ee O 
Then, we solve the problem: 

sol = Minimize[{obj, sup, dem, non}, Flatten[vars]] 

{430, {X1,1 > 15, x1,2 > 0, *1,3 > 25, x2,1 > 20, 

X99. 40, XQ 30> O}} 
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Appendix 


Introduction to Mathematica 


1. Executing commands: 


* To doacalculation with Mathematica, write the command into the notebook document (possibly 
with the help of a palette) and then execute it by pressing the Enter key in the numeric keypad or 
by holding down the Shift key and then pressing the Return key. 


2. Arithmetic: 


* For addition, subtraction, division, and power, use +, -, /, and *. 


* For multiplication, usually the space key is used, but the * key can also be used. For example, 
a bora*b isa product but ab is a single variable. 


3. Useful techniques: 
* To give a name for a result, use =; forexample,a = 17°2. 
* To clear the value of a name, use =.; for example,a =.. 
* To refer to the last result, use %; for example, % + 18°2. 


* Ifan input is ended with a semicolon (a = 17°2;), the input is processed (after pressing, say, 
Enter), but the result is not shown in the notebook. 


¢ Input and output labels of the form In[n] and Out[] can be turned off with the menu command 
Kernel > Show In/Out Names. 


4. Important conventions: 
* All built-in names start with a capital letter. 
* The argument of functions and commands are given within square brackets [ ]. 
* Parentheses (_ ) are only used for grouping terms in expressions. 
5. Constants and mathematical functions: 
° Pi (3.141...), E(2.718...), I (./—1), Infinity. 


* Sqrt[z] or z* (1/2), Exp[z] or E*z, Log[z] (the natural logarithm), Log[b, 2] (a 
logarithm to base b), Abs [z], Sin[z], ArcSin[z],n!,Binomial[n, m]. 


6. Lists: 


* A list is formed with curly braces { };anexample:v = {4, 7, 5}.Alistis simply an ordered 
collection of elements. Lists can be nested, for examplem = {{4, 7, 5}, {2, 6, 3}}. 


* Parts of lists (and of more general expressions) can be picked up with double square brackets 
[{£ 11. For example,m[[2]] gives{2, 6, 3}andm[[2, 1]] gives 2. 


7. Useful commands: 
* A decimal value can be obtained using N[expr]. 
* A decimal value to n digit precision can be obtained using N[expr, nJ. 
* Numbers near to zero can be set to 0 with Chop [expr]. 
* An expression can be simplified with Simplify [expr]. 
* An expression can be expanded out with Expand [expr]. 


¢ All commands with a single argument can also be applied with //; e.g.,expr // N means 
N[expr]. 
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8. Replacements: 


° The syntaxexpr /. x — ais used to replace x in expr with a. Here,x — ais a trans- 
formation rule and /. is the replacement operator. The arrow — can be written by typing the 
two characters -> with no space in between; Mathematica transforms -> to a genuine arrow. 


9. Mapping and threading: 


¢ With Map we can map the elements of a list with a function, that is, with Map we can calculate 
the value of a function at points given in a list. For example, Map[£[#] &, {a, b, c}] 
gives {f[a], fb], f£[c]} (a function like £[#] & is called a pure function). If £ is a 
built-in function with one argument, the name of the function suffices. For example, if we want 
to calculate the 2-norms of the rows of a matrix m, we can write Map[Norm[#] &, mJ], but 
also simply Map[Norm, m]. 


* With Thread we can apply an operation to corresponding parts of two lists. For example, 
Thread[{x + y, x — y} == {2, 5}]gives{x + y == 2, x — y == 5}. 


10. Functions and programs: 


¢ The syntax £[x_] := expr is used to define functions. For example, £[x_, y-] := x + 
Sin[y]. 


* In programs, we often use Modu1e to define local variables or With to define local constants. 
The syntax is £[x_] := Module [{local variables}, body] (similarly for With). 


11. Loading packages: 
* A package is loaded with <<. For example, <<LinearAlgebra* MatrixManipulation. 
12. Using saved notebooks: 


* When you open a saved notebook, Mathematica will not automatically process the inputs anew. 
So, if the notebook contains the definitiona = 172 and youalso want to use this definition in 
the new Mathematica session, you have to process the input anew; simply put the cursor anywhere 
in the input and press Enter or Shift-Return. 


Getting Help 


1. Books, documents, and internet addresses: 


* [Wol03]: A complete description of Mathematica 5; for linear algebra, see Sections 1.5.7 (Solving 
Equations), 1.8.3 (Vectors and Matrices), and 3.7 (Linear Algebra). 


* [WR99]: A description of the packages of Mathematica 4. 
* [WR03]: Linear algebra with Mathematica 5; a long Help Browser document. 


* [Sza00]: Linear algebra with Mathematica 3. 


¢ {Rus04]: A general introduction to Mathematica 5. 


* http://library.wolfram.com/infocenter/BySubject/Mathematics/Algebra/LinearAlgebra/: about 
100 documents relating to linear algebra with Mathematica. 


2. Using the Help Browser: 


* To open the Help Browser, select the menu command Help > Help Browser. 


* In the Help Browser, write a command into the input field and press the Go-button, or choose 
from the lists of topics. 
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* In a notebook, write a command like Eigenvalues and then press the Fl key (Windows) or 
Help key (Macintosh) to read the corresponding Help Browser material. 


* Inanotebook, execute a command like ?7Eigenvalues to get a short description of acommand 
and a link to the more complete information in the Help Browser. 


3. Useful material in the Help Browser: 


* The Mathematica Book > A Practical Introduction to Mathematica > Symbolic Mathematics > 
Solving Equations: Section 1.5.7 of [Wol03]. 


* The Mathematica Booker A Practical Introduction to Mathematica Lists > Vectors and Matrices: 
Section 1.8.3 of [Wol03]. 


* The Mathematica Book > Advanced Mathematics in Mathematica > Linear Algebra: Section 3.7 
of [Wol03]. 


* Built-in Functions > Advanced Documentation > Linear Algebra: [WR03]. 


* Built-in Functions > Lists and Matrices > Vector Operations: A list of commands related to 
vectors. 


¢ Built-in Functions > Lists and Matrices > Matrix Operations: A list of commands related to 
matrices. 


¢ Add-ons and Links> Standard Packages > Linear Algebra: Descriptions of linear algebra packages. 


For Users of Earlier Versions 


Some commands presented in this chapter are new in Mathematica 5.0 or 5.1. Below we have collected 
information about how to do similar calculations with earlier versions (mainly with 4.2). 


* ArrayPlot [m]: replace with ListDensityPlot [Reverse [Abs[m]], ColorFunction 

— (GrayLevel[1 — #] &), AspectRatio -—> Automatic, Mesh — False, 
FrameTicks — False]. 

* CholeskyDecomposition: replace with a command of the same name from the 
LinearAlgebra Cholesky package. 

* ConjugateTranspose [m]: replace with Conjugate [Transpose[m] ]. 

* HessenbergDecomposition: replace with Developer HessenbergDecomposition. 

* MatrixRank [ml]: replace with Dimensions [m] [[2]] — Length[NullSpace[m]]. 

* Maximize[{f, cons}, vars]: replace withConstrainedMax[f, cons, vars]. 

* Minimize[{f, cons}, vars]: replace withConstrainedMin[f, cons, vars]. 

* Most [v]: replace with Drop[v, -1]. 

* Norm: Replace with VectorNorm or MatrixNorm from the LAMM package (note: the default is the 
oo-norm). 

* SingularValueDecomposition: replace with SingularValues. 

* SingularValueList [m]: replace with SingularValues [m] [[2]]. 

* Total: Replace with Apply. For vectors use Apply [Plus, vw]. For column sums of matrices, use 
Apply[Plus, mJ]; for rowsums, useApply[Plus, Transpose[m] ]; and for the sum of all 
elements, use Apply [Plus, Flatten[m] ]. 
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74.1 Introduction 


A significant amount of execution time in complicated linear algebraic programs is known to be spent ina 
few low-level operations. Consequently, reducing the overall execution time ofan application program leads 
to the problem of optimizing these low-level operations. Such low-level optimization are highly machine- 
dependent and are a matter for specialists. A separation has, therefore, been made by the computational 
linear algebra community: On the one hand highly efficient, machine-dependent building blocks of linear 
algebra, the BLAS, (basic linear algebra subprograms) are provided on a given computational platform. 
On the other hand, linear algebra programs attempt to make the most use of those computational blocks 
in order to have as good performance as possible across a wide range of platforms. 

The first major concerted effort to achieve agreement on the specification of a set of linear algebra 
kernels resulted in the Level-1 BLAS [LHK79]. The Level-1 BLAS specification and implementation are the 
result of a collaborative project in 1973 through 1977. The Level-1 BLAS were extensively and successfully 
exploited by LINPACK [DBM79], a software package for the solution of dense and banded linear equations 
and linear least squares problems. 

With the advent of vector machines, hierarchical memory machines, and shared memory parallel 
machines, specifications for the Level-2 and -3 BLAS [DDD90a], [DDD90b], [DDH88a], [DDH88b], 
concerned with matrix—vector and matrix—matrix operations, respectively, were drawn up in 1984 through 
1986 and 1987 to 1988. These specifications made it possible to construct new software to utilize the 
memory hierarchy of modern computers more effectively. In particular, the Level-3 BLAS allowed the 
construction of software based upon block-partitioned algorithms, typified by the linear algebra software 
package LAPACK (see Chapter 75). 

In this chapter, we present information about the BLAS. We begin with basic definition, then describe 
the calling sequences, report a number of facts, and finally conclude with a few examples. 


Definitions: 


Two-dimensional column-major format. A two-dimensional array is said to be in column-major format 
when its entries are stored by column. This is the way the Fortran language stores two-dimensional 
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arrays. For example, the declaration double precision A(2,2) stores the entries of the 2-by-2 
two-dimensional array in the one-dimensional order: A(1,1),A(2,1),A(1,2),A(2,2). 
Row-major format. Storage of the entries by row. This is the way C stores two-dimensional arrays. 
Packed format. The packed format is relevant for symmetric, Hermitian, or triangular matrices. Half 
of the matrix (either the upper part or the lower part depending on an UPLO parameter) is stored in a 
one-dimensional array. For example, if UPLO= ' L', the 3-by-3 symmetric matrix A is stored as A(1,1), 
A(2,1),A(3,1),A(2,2),A(3,2),A(3,3)). 

Leading dimension. In column-major format, the leading dimension is the first dimension of a two- 
dimensional array (as opposed to the dimension of the matrix stored in that array). The leading dimension 


of an array is necessary to access its elements. For example, if LDA is the leading dimension of A, and A 
is stored in the two-dimensional column-major format, Aj,; is stored in position i + (j — 1) * LDA. The 
leading dimension of an m-by-n matrix is often m. It enables convenient abstraction to access submatrices 
(see examples). An m-by-n matrix should have LDA > m. 

Increment. The increment of a vector is the number of storage elements from one element to the next. If 


the increment ofa vector is 1, this means that the vector is stored contiguously in memory; an increment of 
2 corresponds to using every other array element. The increment is useful to manipulate rows of a matrix 
stored in column-major format (see Example 3). 

BLAS vector description. A vector description in BLAS is defined by three quantities — a vector length, 
an array or a starting element within an array, and the increment. This gives (n, X, 1) for a vector of size 
n starting at index X with increment of 1. 

BLAS description of a general matrix. A general matrix is described in BLAS by four quantities: a 
TRANS label ('N', 'T', or 'H' ), its dimensions mand n, an array ora starting element within an array, 
and a leading dimension. This gives (TRANS ,M,N,A, LDA) if one wants to operate on the TRANS of an 
M-by-N matrix starting in A(1,1) and stored in an array of leading dimension LDA. 

Prefixes The first letter of the name of a BLAS routine indicates the Fortran type on which it operates. 


S - REAL C - COMPLEX 
D-DOUBLE PRECISION = Z-COMPLEX*16 


Prefixes for Level-2 and Level-3 BLAS 
Matrix types: 


GE - GEneral GB - General Band 
SY-SYmmetric SB - Sym. Band SP - Sym. Packed 
HE-HErmitian HB - Herm. Band HP - Herm. Packed 


TR-TRiangular TB - Triang. Band TP - Triang. Packed 


Level-2 and Level-3 BLAS Options 


TRANS = ‘No transpose’, ‘Transpose’, ‘Conjugate transpose’ 
UPLO = ‘Upper triangular’, “Lower triangular’ 

DIAG = ‘Non-unit triangular’, ‘Unit triangular’ 

SIDE = ‘Left’ ‘Right’ on the right) 


Calls: 


1. Below is the calling sequence for the GEMV matrix-vector multiply subroutine. The GEMV routine 
performs the mathematical operation 


y<—aAx+fBy, or y <— aA'x + By. 


BLAS 


Facts: 


74-3 


For double precision, the calling sequence looks like 


SUBROUTINE DGEMV ( TRANS, M, N, ALPHA, A, LDA, X, INCX, 
$ BETA, Y, INCY ) 


The types of the variables are as follows: 


DOUBLE PRECISION ALPHA, BETA 

INTEGER INCX, INCY, LDA, M, N 
CHARACTER * 1 TRANS 

DOUBLE PRECISION A( LDA, * ), X( * ), Y( * ) 


The meaning of the variables is as follows: 

TRANS : Specifies the operation to be performed as 
TRANS='N' y < aAx + By, 
TRANS='T' y —a@A?x-+ By. 


M: Specifies the number of rows of the matrix A. 
N : Specifies the number of columns of the matrix A. 
ALPHA: Specifies the scalar a. 


A: Points to the first entry of the two-dimensional array that stores the elements of A in 
column major format. 


LDA : Specifies the leading dimension of A. 


X : Points to the incremented array that contains the vector x. 


INCX : Specifies the increment for the elements of X. 
BETA : Specifies the scalar B. 


Y : Points to the incremented array that contains the vector y. 


INCY : Specifies the increment for the elements of Y. 


All variables are left unchanged after completion of the routines, except Y. 


. The following table is the quick reference to the BLAS. 


. The BLAS represent fairly simple linear algebra kernels that are easily coded in a few lines. One could 


also download reference source codes and compile them (http: //www.netlib.org/blas). 
However, this strategy is unlikely to give good performance, no matter the level of sophistication of 
the compiler. The recommended way to obtain an efficient BLAS is to use vendor BLAS libraries, 
or to install the ATLAS [WPD01] package. Figure 74.1 illustrates the fact that ATLAS BLAS clearly 
outperform the reference implementation for the matrix—matrix multiply by a factor of five on a 
modern architecture. 

To a great extent, the user community has embraced the BLAS, not only for performance reasons, 
but also because developing software around a core of common routines like the BLAS is good 
software engineering practice. Highly efficient, machine-specific implementations of the BLAS are 
available for most modern high-performance computers. To obtain an up-to-date list of available 
optimized BLAS, see the BLAS FAQ at http: //www.netlib.org/blas. 

Level-1 BLAS operates on O(n) data and performs O(n) operations. Level-2 BLAS operates on 
O(n*) data and performs O(n’) operations. Level-3 BLAS operates on O(n?) data and performs 
O(n?) operations. Therefore, the ratio operation/data is O(1) for Level-1 BLAS and Level-2 BLAS, 
but O(n) for Level-3 BLAS. On modern architecture, where memory access is particularly slow 
compared to computation time, Level-3 BLAS exploit this O() ratio to mask the memory access 
bottleneck (latency and bandwidth). Figure 74.1 illustrates the fact that Level-3 BLAS on a modern 
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FIGURE 74.1 Performance in GFlops of four BLAS routines for two different BLAS libraries on an Intel Xeon 
CPU running at 3.20 GHz. The first BLAS library represents an optimized BLAS library (here, ATLAS v3.7.8), its 
four performance curves are given with a solid line on the graph. The second library represents straightforward 
implementation (here reference BLAS from netlib); its four performance curves are given with a dotted line on the 
graph. This graph illustrates two facts: An optimized Level-3 BLAS is roughly five times faster than Level-1 or Level-2 
BLAS and it is also roughly five times faster than a reference Level-3 BLAS implementation. To give an idea of the actual 
time, multiplying two 6000-by-6000 matrices on this machine will take about 2 minutes using the ATLAS BLAS while 
it will take about 10 minutes using the reference BLAS implementation. 


architecture performs 5 times more floating-point operations per second than a Level-2 or Level-1 
BLAS routine. Most of the linear algebra libraries try to make as much as possible use of Level-3 
BLAS. 

4. Most of shared memory computers have a multithreaded BLAS library. By programming a sequen- 
tial code and linking with the multithreaded BLAS library, the application will use all the computing 
units of the shared memory system without any explicit parallelism in the user application code. 

5. Although here we present only calling sequences from Fortran, it is possible to call the BLAS from 
C. The major problem for C users is the Fortran interface used by the BLAS. The data layout 
(the BLAS interface assumes column-major format) and the passage of parameters by reference 
(as opposed to values) has to be done carefully. See Example 2 for an illustration. Nowadays most 
BLAS distributions provide a C-interface to the BLAS. This solves these two issues and, thus, we 
highly recommend its use. 

6. The Level-1, Level-2, and Level-3 BLAS are now extended by a new standard with more functional- 
ity and better software standard; see [BDD02], [Don02]. For example, the C-interface to the BLAS 
(see Fact 5) is included in this new BLAS. 


Examples: 


In all of these examples, A is an m-by-n matrix and it is stored in an M-by-N array starting at position A, 


B is an n-by-n matrix and it is stored in an N-by-N array starting at position B, C is an m-by-m matrix 
and it is stored in an M-by-M array starting at position C, X is a vector of size n and it is stored in an N array 
starting at position X, and Y is a vector of size m and it is stored in an M array starting at position Y. All 
the two-dimensional arrays are in column-major format, all the one-dimensional arrays have increment 
one. We assume that m and n are both greater than 11 in Example 3. 


BLAS 74-7 


1. To perform the operation y <— Ax the BLAS calling sequence is 


CALL DGEMV ( 'N', M, N, 1.0D0, A, M, X, 1, 0.0D0, Y, 1 ) 


2. To perform the operation x <— aA! y + fx the BLAS calling sequence is 


CALL DGEMV ( 'T', M, N, ALPHA, A, M, Y, 1, BETA, X, 1 ) 


From C, this would give 


IONE = 1; 
dgemv ( "T", &M, &N, &ALPHA, A, &M, Y, &IONE, &BETA, X, &IONE ); 


3. To perform the operation 
y(2:10) <— 2A(3: 11,4: 11) * B(4,3: 10)" — 3y(2: 10); 
the BLAS calling sequence is 


CALL DGEMV ( 'N', 9, 8, 2.0D0, A(3,4), M, B(4,3), N, -3.0D0, 
¥(2), 1) 


(Note the use of LDA to operate on the submatrix of A and the use of INCX to operate ona rowof B.) 

4. LAPACK (see Chapter 75) is a library based on the BLAS. Opening its Fortran files enables one to 
gain a good understanding of how to use BLAS routines. For example, a good way to start is to have 
alook at dget rf . £, which performs a right-looking LU factorization by making calls to DTRSM 
and DGEMM. 
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75.1 Introduction 


LAPACK (linear algebra package) is an open source library of programs for solving the most commonly 
occurring numerical linear algebra problems [LUG99]. Original codes of LAPACK are written in Fortran 
77. Complete documentation as well as source codes are available online at the Netlib repository [LAP]. 
LAPACK provides driver routines for solving complete problems such as linear equations, linear least 
squares problems, eigenvalue problems, and singular value problems. Each driver routine calls a sequence 
of computational routines, each of which performs a distinct computational task. In addition, LAPACK 
provides comprehensive error bounds for most computed quantities. LAPACK is designed to be portable 
for sequential and shared memory machines with deep memory hierarchies, in which most performance 
issues could be reduced to providing optimized versions of the Basic Linear Algebra Subroutines (BLAS). 
(See Chapter 74). 

There have been a number of extensions of LAPACK. LAPACK95 is a Fortran 95 interface to the 
Fortran 77 LAPACK [LAP95]. CLAPACK and JLAPACK libraries are built using the Fortran to C (f2c) and 
Fortran to Java (f2j) conversion utilities, respectively [CLA], [JLA]. LAPACK++ is implemented in C++ and 
includes a subset of the features in LAPACK with emphasis on solving linear systems with nonsymmetric 
matrices, symmetric positive definite systems, and linear least squares systems [LA+]. ScaLAPACK is a 
portable implementation of some of the core routines in LAPACK for parallel distributed computing [Sca]. 
ScaLAPACK is designed for distributed memory machines with very powerful homogeneous sequential 
processors and with homogeneous network interconnections. 

The purpose of this chapter is to acquaint the reader with 10 essential numerical linear algebra problems 
and LAPACK’s way of solving those problems. The reader may find it helpful to consult Chapter 74, where 
some of the terms used here are defined. The following table summarizes these problems and sections that 
are treated in version 3.0 of LAPACK. 
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Type of Problem Acronyms Section 
Linear system of equations SV 75.2 
Linear least squares problems LLS 75.3 
Linear equality-constrained least squares problem LSE 75.4 
General linear model problem GLM 75.5 
Symmetric eigenproblems SEP 75.6 
Nonsymmetric eigenproblems NEP 75.7 
Singular value decomposition SVD 75.8 
Generalized symmetric definite eigenproblems GSEP 75.9 
Generalized nonsymmetric eigenproblems GNEP 75.10 
Generalized (or quotient) singular value decomposition GSVD (QSVD) 75.11 


Sections have been subdivided into the following headings: (1) Definition: defines the problem, 
(2) Background: discusses the background of the problem and references to the related sections in this 
handbook, (3) Driver Routines: describes different types of driver routines available that solve the same 
problem, (4) Example: specifies the calling sequence for a driver routine that solves the problem followed 
by numerical examples. 

All LAPACK routines are available in four data types, as indicated by the initial letter “x” of each 
subroutine name: x = “S” means real single precision, x = “D”, real double precision, x = “C”, complex 
single precision, and x = “Z”, complex*16 or double complex precision. In single precision (and complex 
single precision), the computations are performed with a unit roundoff of 5.96 x 10~°. In double precision 
(and complex double precision) the computations are performed with a unit roundoff of 1.11 x 107!°. 

All matrices are assumed to be stored in column-major format. The software can also handle submatrices 
of matrices, even though these submatrices may not be stored in consecutive memory locations. For 
example, to specify the 10—by—10 submatrix lying in rows and columns 11 through 20 of a 30-by—30 
matrix A, one must specify 


* A(11, 11), the upper left corner of the submatrix 


* 30 = Leading dimension of A in its declaration (often denoted LDA in calling sequences) 


* 10 = Number of rows of the submatrix (often denoted M, can be at most LDA) 
* 10 = Number of columns of submatrix (often denoted N) 


All matrix arguments require these 4 parameters (some subroutines may have fewer inputs if, for example, 
the submatrix is assumed square so that M = N). (See Chapter 74, for more details.) 

Most of the LAPACK routines require the users to provide them a workspace (WORK) and its dimension 
(LWORK). The optimal workspace dimension refers to the workspace dimension, which enables the code to 
have the best performance on the targeted machine. The computation of the optimal workspace dimension 
is often complex so that most of LAPACK routines have the ability to compute it. If a LAPACK routine 
is called with LWORK=-1, then a workspace query is assumed. The routine only calculates the optimal 
size of the WORK array and returns this value as the first entry of the WORK array. If a larger workspace 
is provided, the extra part is not used, so that the code runs at the optimal performance. A minimal 
workspace dimension is provided in the document of routines. If a routine is called with a workspace 


dimension smaller than the minimal workspace dimension, the computation cannot be performed. 
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Definitions: 


The problem of linear equations is to compute a solution X of the system of linear equations 
anna (75.1) 


where A is an n—by—n matrix and X and B are n—by—m matrices. 
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Backgrounds: 


The theoretical and algorithmic background of the solution of linear equations is discussed extensively in 
Chapter 37 through Chapter 41, especially Chapter 38. 


Driver Routines: 


There are two types of driver routines for solving the systems of linear equations — simple driver and 
expert driver. The expert driver solves the system (Equation 75.1), allows A be replaced by A’ or A*; and 
provides error bounds, condition number estimate, scaling, and can refine the solution. Each of these types 
of drivers has different implementations that take advantage of the special properties or storage schemes 
of the matrix A, as listed in the following table. 


Routine Names 


Data Structure (Matrix Storage Scheme) Simple Driver Expert Driver 
General dense xGESV xGESVX 
General band xGBSV xGBSVX 
General tridiagonal xGTSV xGTSVX 
Symmetric/Hermitian positive definite xPOSV xPOSVX 
Symmetric/Hermitian positive definite (packed storage) xPPSV xPPSVX 
Banded symmetric positive definite xPBSV xPBSVX 
Tridiagonal symmetric positive definite xPTSV xPTSVX 
Symmetric/Hermitian indefinite xSYSV/xHESV — xSYSVX/xHESVX 
Symmetric/Hermitian indefinite (packed storage) xSPSV/xHPSV —- xSPSVX/xHPSVX 
Complex symmetric CSYSV/ZSYSV_ — CSYSVX/ZSYSVX 


The prefixes GE (for general dense), GB (for general band), etc., have standard meanings for all the 
BLAS and LAPACK routines. 


Examples: 


Let us show how to use the simple driver routine SGESV to solve a general linear system of equations. 
SGESV computes the solution of a real linear Equation 75.1 in single precision by first computing the LU 
decomposition with row partial pivoting of the coefficient matrix A, followed by the back and forward 
substitutions. SGESV has the following calling sequence: 


CALL SGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO ) 


Input to SGESV: 
N: The number of linear equations, i.e., the order of A.N > 0. 
NRHS: The number of right-hand sides, i.e., the number of columns of B. NRHS > 0. 


A, LDA: The N-by-N coefficient matrix A and the leading dimension of the array A. 
LDA > max(1,N). 


B, LDB: TheN—by—NRHS matrix B and the leading dimension ofthe array B. LDB > max(1,N). 


Output from SGESV: 


A: The factors L and U from factorization A = P LU; the unit diagonal elements of L are 
not stored. 


IPIV: The pivot indices that define the permutation matrix P; row i of the matrix was 
interchanged with row IPIV(z). 


B: If INFO = 0, the N-by-NRHS solution X. 


INFO: = 0, successful exit. If INFO = —j, the jth argument had an illegal value. If 
INFO = j, U(j,j) is exactly zero. The factorization has been completed, but the 
factor U is singular, so the solution could not be computed. 
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Consider a 4—by—4 linear system of Equation (75.1), where 


> axe. & 23 
Bo VO BZ 32 
Bie or Seg gn ils 
5 7 9 10 31 


T 
The exact solution is x = 1 11 1| . Upon calling SGESV, the program successfully exits with INFO = 0 
and the solution X of (75.1) resides in the array B 


0.9999998 
1.0000004 
0.9999998 
1.0000001 


Since SGESV performs the computation in single precision arithmetic, it is normal to have an error of 
the order of 10~° in the solution X. By reading the lower diagonal entries in the array A and filling the 
diagonal entries with ones, we recover the lower unit triangular matrix L of the LU factorization with row 
partial pivoting of A as follows: 


1 0 0 0 

_ |0.8571429 1 0 0 
0.7142857 0.2500000 1 0 
0.7142857 0.2500000 —0.2000000 1 


The upper triangular matrix U is recovered by reading the diagonal and upper diagonal entries in A. That is: 


7.0000000 10.0000000 8.0000000 7.0000000 
QO —0.5714293 3.1428566 2.9999995 
0 0 2.5000000 4.2500000 
0 0 0 0.1000000 


Finally, the permutation matrix P is the identity matrix that exchanges its ith row with row IPIV(i), for 
i=n,...,1. Since 


piv = [2 3 4 4], 


we have 


oor oO 
oro°eo 
ep OOO 
oOo oF 


75.3 Linear Least Squares Problems 


Definitions: 


The linear least squares (LLS) problem is to find 
min ||b — Ax||>, (75.2) 


where A is an m—by-—n matrix and b is an m element vector. 
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Backgrounds: 


The most usual case is m > n and rank(A) = n. In this case, the solution to the LLS problem (75.2) is 
unique, and the problem is also referred to as finding a least squares solution to an overdetermined system 
of linear equations. When m < n and rank(A) = m, there are an infinite number of solutions x that 
exactly satisfy b — Ax = 0. In this case, it is often useful to find the unique solution x that minimizes ||x||2, 
and the problem is referred to as finding a minimum norm solution to an underdetermined system of linear 
equations. (See Chapter 5.8 and Chapter 39 for more information on linear least squares problems.) 


Driver Routines: 


There are four types of driver routines that solve the LLS problem (75.2) and also allow A be replaced by 
A*. In the general case when rank(A) < min(m,n), we seek the minimum norm least squares solution x 
that minimizes both ||x||2 and |/b — Ax||2. The types of driver routines are categorized by the methods 
used to solve the LLS problem, as shown in the following table. 


Type of Matrix Algorithm Routine Names 
General dense QR or LQ factorization xGELS 
General dense | Complete orthogonal factorization xGELSY 
General dense SVD xGELSS 
General dense — Divide-and-conquer SVD xGELSD 


xGELSD is significantly faster than xGELSS, but it demands somewhat more workspace depending 
on the matrix dimensions. Among these routines, only xGELS requires A to be full rank while xGELSY, 
xGELSS, and xGELSD allow less than full rank. 

Note that all driver routines allow several right-hand side vectors b and corresponding solutions x to 
be handled in a single call, storing these vectors as columns of matrices B and X, respectively. However, 
the LLS problem (75.2) is solved for each right-hand side independently; that is not the same as finding a 
matrix X which minimizes || B — AX||2. 


Examples: 


Let us show how to use the simple driver routine SGELS to solve the LLS problem (75.2). SGELS computes 
the QR decomposition of the matrix A, updates the vector b, and then computes the solution x by back 
substitution. SGELS has the following calling sequence: 


CALL SGELS( TRANS, M, N, NRHS, A, LDA, B, LDB, WORK, LWORK, INFO ) 


Input to SGELS: 
TRANS: = 'N' or 'T': solves the LLS with A or A’. 
, N: The numbers of rows and columns of the matrix A.M > 0andN > 0. 
NRHS: The number of rows and columns of the matrices B and X.NRHS > 0. 
, LDA: TheM—by—N matrix A and theleading dimension ofthe array A,LDA > max(1,M). 


BP 8 8 


, LDB: The matrix B and the leading dimension of the array B, LDB > max(1,M,N). 
IfTRANS = 'N',then BisM—by-NRHS.IfTRANS = 'T', then B isN—-by-NRHS. 
WORK, LWORK: Theworkspacearrayand its dimension. LWORK > min(M, N) + max(1,M, N, NRHS). 
If LWORK = —1, then a workspace query is assumed; the routine only calculates the op- 
timal size of the WORK array, and returns this value as the first entry of the WORK array. 


Output from SGELS: 


B: It is overwritten by the solution vectors, stored columnwise. 


¢ If TRANS = 'N' andM = N, rows 1 to N of B contain the solution vectors of the LLS 
problem min, ||b — Ax||2; the residual sum of squares in each column is given by the 
sum of squares of elements N + 1 to M in that column; 
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[TRANS = 'N' andM <N, rows 1 to N of B contain the minimum norm solution 


vectors of the underdetermined system AX = B; 


¢ If TRANS = 'T' andM => N, rows 1 to M of B contain the minimum norm solution 
vectors of the underdetermined system A'X=B; 

¢ If TRANS = 'T' and M <N, rows 1 to M of B contain the solution vectors of the 
LLS problem min, ||b — A’x||2; the residual sum of squares for the solution in each 


col 


umn is given by the sum of the squares of elements M+1 to N in that column. 


WORK: If INFO = 0, WORK(1) returns the optimal LWORK. 


INFO: INFO = 0 if successful exit. If INFO = —j, the jth input argument had an illegal 
value. 


Consider an LLS problem (75.2) with a 6-by—5 matrix A and a 6-by—1 matrix b: 


ae a ¢ 8 51 
14-69 21 2 0 = 
66.29! Sige 2256 

Be aye ee aay Egy. 4) eB ee 
Bi Beate ede 10 
4-54 OH OG =19 


T 
The exact solution of the LLS problemisx = ]1 2 —1 3 —4| with residual ||b — Ax||, = 0. Upon 


calling SGELS, the first 5 elements of B are overwritten by the solution vector x: 


ie 

2.0000196 

x = | —0.9999972| , 
3.0000386 
—4,0000405 


while the sixth element of B contains the residual sum of squares 0.0000021. With M = 6,N = 5, NRHS = 1, 


LWORK has been set to 11. For such a small matrix, the minimal workspace is also the optimal workspace. 


75.4 The 


Linear Equality-Constrained Least Squares Problem 


Definitions: 


The linear equality-constrained least squares (LSE) problem is 


min ||c — Ax||, subjectto Bx=d, (75.3) 


where A is an m—by—n matrix and B is a p—by—n matrix, c is an m-vector, and d is a p-vector, with 


pinsme p. 


Backgrounds: 


Under the assumptions that B has full row rank p and the matrix A has full column rank n, the LSE 


problem (75.3) has a unique solution x. 
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Driver Routines: 


The driver routine for solving the LSE is xGGLSE, which uses a generalized QR factorization of the matrices 
Aand B. 


Examples: 


Let us show how to use the driver routine SGGLSE to solve the LSE problem (75.3). SGGLSE first computes 
a generalized QR decomposition of A and B, and then computes the solution by back substitution. SGGLSE 
has the following calling sequence: 


CALL SGGLSE( M, N, P, A, LDA, B, LDB, C, D, X, WORK, LWORK, INFO ) 


Input to SGGLSE: 


M, P: The numbers of rows of the matrices A and B, respectively. M > OandP > 0. 

N: The number of columns of the matrices Aand B.N > 0.NotethatO < P < N < 
M+P. 

A, LDA: The M—by-N matrix A and the leading dimension of the array A. LDA > 


max(1,M). 


B, LDB: The P—by-N matrix B and the leading dimension of the array B. LDB > 
max(1, P). 


C, D: The right-hand side vectors for the least squares part, and the constrained equation 
part of the LSE, respectively. 


WORK, LWORK: The workspace array and its dimension. LWORK > max(1,M+-N+ P). 


If LWORK = —1, then a workspace query is assumed; the routine only calculates the 
optimal size of the WORK array, and returns this value as the first entry of the WORK 
array. 


Output from SGGLS! 


v 


C: The residual sum of squares for the solution is given by the sum of squares of elements 
N-P+1 to M of vector C. 


X: The solution of the LSE problem. 
WORK: If INFO = 0,WORK(1) returns the optimal LWORK. 


INFO: =0 if successful exit. If INFO = —j, the jth argument had an illegal value. 


Let us demonstrate the use of SGGLSE to solve the LSE problem (75.3), where 


1 1 1 
t 34 ot? 4 2 7 
BP NG te ae =|} 1 Ei aama in a=] 
ft! “ae a 4 


The unique exact solution is x = +[46 —2 12]". Upon calling SGGLSE with this input data and 
M = 4,N = 3,P = 2, LWORK = 9, an approximate solution of the LSE problem is returned in X: 


X = [5.7500000 —0.2500001 1.4999994]". 
The array C is overwritten by the residual sum of squares for the solution: 


C = [4.2426405 8.9999981 2.1064947 0.2503501]". 
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75.5 A General Linear Model Problem 


Definitions: 


The general linear model (GLM) problem is 
min |ly||2 subjectto d= Ax-+ By, (75.4) 
XY 

where A is an n—by—m matrix, B is an n—by—p matrix, and d is a n-vector, with m <n <m-4 p. 


Backgrounds: 


When B = I, the problem reduces to an ordinary linear least squares problem (75.2). When B is square 
and nonsingular, the GLM problem is equivalent to the weighted linear least squares problem: 


min || B~!(d — Ax)|l>. 


Note that the GLM is equivalent to the LSE problem 


o-|o 1] ;| 


Therefore, the GLM problem has a unique solution of the matrix 


min 
xY 


subject to [A B| A =d. 


2 


i and has full column rank m+ p. 


Driver Routines: 


The driver routine for solving the GLM problem (75.4) is xGGGLM, which uses a generalized QR factor- 
ization of the matrices A and B. 


Examples: 


Let us show how to use the driver routine SGGGLM to solve the GLM problem (75.4). SGGGLM computes 
a generalized QR decomposition of the matrices A and B, and then computes the solution by back 
substitution. SGGGLM has the following calling sequence: 


CALL SGGGLM( N, M, P, A, LDA, B, LDB, D, X, Y, WORK, LWORK, INFO ) 


Input to SGGGLM: 


N: The number of rows of the matrices Aand B.N > 0. 


M, P: The number of columns of the matrices A and B, respectively. 0 < M < Nand 


P > N-M. 

A, LDA: The N-by-M matrix A and the leading dimension of the array A. LDA > 
max(1,N). 

B, LDB: The N-by—P matrix B and the leading dimension of the array B. LDB > 
max(1,N). 

D: The left-hand side of the GLM equation. 

WORK, LWORK: The workspace array and its dimension. LWORK > max(1,N+M+P). 
If LWORK = -1, then a workspace query is assumed; the routine only calculates the 


optimal size of the WORK array, and returns this value as the first entry of the WORK 
array. 
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Output from SGGGLM: 


X, Y: Solution vectors. 
WORK: If INFO = 0,WORK(1) returns the optimal LWORK. 
INFO: INFO = 0 ifsuccessful exit. If INFO = —j, the jth argument had an illegal value. 


Let us demonstrate the use of SGGGLM for solving the GLM problem (75.4), where 


1 2 1 4 1 2 2 7.99 

—1 1 1 1 —l1 1 —2 0.98 

Acs pod 2) =] 1}, B= 3 1 6], d= }—2.98 
—l1 2 -1 -l Dea 2, 4 3.04 

1 1 1 2 1 -l 2 4.02 


Upon calling SGGGLM with this input data, N = 5,M = 4,P = 3, LWORK = 12, the program successfully 
exits and returns the following solution vectors: 


X = [1.002950 2.001435 —0.987797 0.009080] 7 
and 


Y = [0.003435 —0.004417 0.006871]’. 


75.6 Symmetric Eigenproblems 


Definitions: 
The symmetric eigenvalue problem (SEP) is to find the eigenvalues, A, and corresponding eigenvectors, 
x # 0, such that 

Ax = Xx, (75.5) 


where A is real and symmetric. If A is complex and Hermitian, i.e., A* = A, then it is referred to as the 
Hermitian eigenvalue problem. 


Backgrounds: 


When all eigenvalues and eigenvectors have been computed, we write 
A= XAX"*, (75.6) 


where A is a diagonal matrix whose diagonal elements are real and are the eigenvalues, and X is an orthog- 
onal (or unitary) matrix whose columns are the eigenvectors. This is the classical spectral decomposition of 
A. The theoretical and algorithmic backgrounds is of the solution of the symmetric eigenvalue problem 
are discussed in Chapter 42. 


Driver Routines: 


There are four types of driver routines for solving the SEP (75.5) and each has its own variations that take 
advantage of the special structure or storage of the matrix A, as summarized in the following table. 
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Types of Matrix Routine Names 
(Storage Scheme) Simple Driver  Divide-and-Conquer Expert Driver RRR Driver 
General symmetric xSYEV xSYEVD xSYEVX xSYEVR 
General symmetric 
(packed storage) xSPEV xSPEVD xSPEVX - 
Band matrix xSBEV xSBEVD xSBEVX - 
Tridiagonal matrix xSTEV xSTEVD xSTEVX xSTEVR 


The simple driver computes all eigenvalues and (optionally) eigenvectors. The expert driver computes 
all or a selected subset of the eigenvalues and (optionally) eigenvectors. The divide-and-conquer driver 
has the same functionality as, yet outperforms, the simple driver, but it requires more workspace. The 
relative robust representation (RRR) driver computes all or a subset of the eigenvalues and (optionally) 
the eigenvectors. The last one is generally faster than any other types of driver routines and uses the least 
amount of workspace. 


Examples: 


Let us show how to use the simple driver SSYEV to solve the SEP (75.5) by computing the spectral 
decomposition (75.6). SSYEV first reduces A to a tridiagonal form, and then uses the implicit QL or QR 
algorithm to compute eigenvalues and optionally eigenvectors. SSYEV has the following calling sequence: 


CALL SSYEV( JOBZ, UPLO, N, A, LDA, W, WORK, LWORK, INFO ) 


Input to SSYEV: 


JOBZ: = 'N', compute eigenvalues only; 


= 'V', compute eigenvalues and eigenvectors. 


UPLO: = 'U', the upper triangle of A is stored in the array A;ifUPLO = 'L', the lower 
triangle of A is stored. 


N: The order of the matrix A.N > 0. 


A, LDA: The symmetric matrix A and the leading dimension of the array A. LDA > 
max(1,N). 


WORK, LWORK: The workspace array and its dimension. LWORK > max(1,3 * N— 1). 


If LWORK = —1, then a workspace query is assumed; the routine only calculates the 
optimal size of the WORK array, and returns this value as the first entry of the WORK 
array. 


Output from SSYEV: 


A: The orthonormal eigenvectors X,if JOBZ = 'V'. 
W: The eigenvalues A in ascending order. 
WORK: If INFO = 0,WORK(1) returns the optimal LWORK. 


INFO: = 0 if successful exit. If INFO = —j, the jth input argument had an illegal value. 
If INFO = j, the j off-diagonal elements of an intermediate tridiagonal form did not 
converge to zero. 


Let us demonstrate the use of SSYEV to solve the SEP (75.5), where 


ee kU 
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The exact eigenvalues are 1, 2, 5, and 10. Upon calling SSYEV with the matrix A andN = 4, LWORK = 3 * N— 
1 = 11, Ais overwritten by its orthonormal eigenvectors X. 


0.7071068 —0.0000003 0.3162279 0.6324555 
—0.7071068 0.0000001 0.3162278 0.6324555 
0.0000002 0.7071069 —0.6324553 0.3162278 
—0.0000001 —0.7071066 —0.6324556 0.3162278 


The eigenvalues that correspond to the eigenvectors in each column of X are returned in W: 
W = [0.9999996 1.9999999  4.9999995 10.0000000]. 


75.7 Nonsymmetric Eigenproblems 


As is customary in numerical linear algebra, in this section the term left eigenvector of A means a (column) 
vector y such that y* A = Ay”. This is contrary to the definition in Section 4.3, under which y* would be 
called a left eigenvector. 


Definitions: 
The nonsymmetric eigenvalue problem (NEP) is to find the eigenvalues, A, and corresponding (right) 
eigenvectors, x # 0, such that 

Ax = hx (75.7) 


and, perhaps, the left eigenvectors, y 4 0, satisfying 
yA=hy*. (75.8) 


Backgrounds: 
The problem is solved by computing the Schur decomposition of A, defined in the real case as 
A= ZTZ", 


where Z is an orthogonal matrix and T is an upper quasi-triangular matrix with 1-by—1 and 2—by-2 
diagonal blocks, the 2—by—2 blocks corresponding to complex conjugate pairs of eigenvalues of A. In the 
complex case, the Schur decomposition is 

A=ZTZ*, 


where Z is unitary and T is a complex upper triangular matrix. 

The columns of Z are called the Schur vectors. For each k (1 < k < n), the first k columns of Z form 
an orthonormal basis for the invariant subspace corresponding to the first k eigenvalues on the diagonal 
of T. It is possible to order the Schur factorization so that any desired set of k eigenvalues occupies the k 
leading positions on the diagonal of T. The theoretical and algorithmic background of the solution of the 
nonsymmetric eigenvalue problem is discussed in Chapter 43. 


Driver Routines: 

Both the simple driver xGEEV and expert driver xGEEVX are provided. The simple driver computes all 
the eigenvalues of A and (optionally) the right or left eigenvectors (or both). The expert driver performs 
the same task as the simple driver plus the additional feature that it balances the matrix to try to improve 
the conditioning of the eigenvalues and eigenvectors, and it computes the condition numbers for the 
eigenvalues or eigenvectors (or both). 


Examples: 


Let us show how to use the simple driver SGEEV to solve the NEP (75.7). SGEEV first reduces A to an upper 
Hessenberg form (a Hessenberg matrix is a matrix where all entries below the first lower subdiagonal are 


75-12 Handbook of Linear Algebra 


zeros), and then uses the implicit QR algorithm to compute the Schur decomposition, and finally computes 
eigenvectors of the upper quasi-triangular matrix. SGEEV has the following calling sequence: 


CALL SGEEV( JOBVL, JOBVR, N,A, LDA,WR, WI, VL, LDVL, VR, LDVR, WORK, 
LWORK, INFO ) 


Input to SGEEV: 


JOBVL, JOBVR: = 'V', the left and/or right eigenvectors are computed; 


= 'N', the left and/or right eigenvectors are not computed. 
N: The order of the matrix A.N > 0. 


A, LDA: The matrix A and the leading dimension of the array A. LDA > max(1,N). 


LDVL, LDVR: The leading dimensions of the arrays VL and VR if the left and right 
eigenvectors are computed. LDVL, LDVR > N. 


WORK, LWORK: Theworkspace array and its dimension. LWORK > max(1, 3 * N). Ifeigen- 
vectors are computed, LWORK > 4 * N. For good performance, LWORK must generally 


be larger. 

If LWORK = —1, then a workspace query is assumed; the routine only calculates the 
optimal size of the WORK array, and returns this value as the first entry of the WORK 
array. 


Output from SGEEV: 


WR, WI: The real and imaginary parts of the computed eigenvalues. Complex conju- 
gate pairs of eigenvalues appear consecutively with the eigenvalue having the positive 
imaginary part first. 


VL: Ifthe jth eigenvalue A; is real, then the jth left eigenvector y; is stored in VL(:, j). 
If the jth and (j + 1)-st eigenvalues A; and 4,4; form a complex conjugate pair, 
then VL(:, 7) +7-VL(:, 7 + 1) and VL(:, 7) —i-VL(:, j + 1) are the corresponding left 
eigenvectors y; and yj41. 


VR: Ifthe jth eigenvalue A; is real, then the jth right eigenvector x; is stored in VR(:, j). If 
the jth and (j + 1)-st eigenvalues A; and Aj; form a complex conjugate pair, then 
vR(:, 7) +72 - VRC: 7 + 1) and VR(:, 7) — i - VR(:, j + 1) are the corresponding right 
eigenvectors x; and xj+1. 


WORK: If INFO = 0,WORK(1) returns the optimal LWORK. 


INFO: = 0 if successful exit. If INFO = —j, the jth argument had an illegal value. If 
INFO = j,the QRalgorithm failed to compute all the eigenvalues, and no eigenvectors 
have been computed; elements j + 1 : NofWRand WI contain eigenvalues, which have 


converged. 


Let us demonstrate the use of SGEEV for solving the NEP (75.7), where 


wuow 
a 
| 
Ww 
| 
ul 


The exact eigenvalues are 12,1 +7 -5,1 —1i- 5, and 2. Upon calling SGEEV with this matrix and 
N = 4, LWORK = 4 * N = 16, each eigenvalue, A, is retrieved by combining the jth entry in WR and WI. 
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such that A; = WR(j) +i -WI(j). IfWI(j) is 0, then the jth eigenvalue is real. For this example, we have 


A, = 12.0000000 
Az = 1.000000 + 7 - 5.0000005 
A3 = 1.000000 — 7 - 5.0000005 
A4g = 1.9999999. 


The left eigenvectors are stored in VL. Since the first and fourth eigenvalues are real, their eigenvectors 
are the corresponding columns in VL, that is, y) = VL(:,1) and yy = VL(:,4). Since the second and 
third eigenvalues form a complex conjugate pair, the second eigenvector, yy = VL(:,2) +7 - VL(:,3) and 
the third eigenvector, y3 = VL(:,2) — i - VL(:,3). If we place all the eigenvectors in a matrix Y where 
Y = [y1, 2, 3, ya], we have 


—0.5000001 0.0000003 — 7 - 0.4999999 0.0000003 + 7 - 0.4999999 0.5000000 

0.4999999 —0.5000002 —0.5000002 0.5000001 
—0.5000000 —0.5000000 — 7 - 0.0000002 —0.5000000 +7 -0.0000002 —0.4999999 
—0.5000001 —0.0000003 +7 -0.5000000 —0.0000003 — i - 0.5000000 0.5000001 


y= 


The right eigenvectors x; can be recovered from VR in the way similar to the left eigenvectors. The right 
eigenvector matrix X is 


—0.5000000 0.5000002 0.5000002 0.5000001 

0.4999999 + —0.0000001 —i-0.5000000 —0.0000001 + 7 - 0.5000000 0.5000000 
—0.5000000 —0.0000001 — 7 - 0.4999999 —0.0000001 +7 - 0.4999999 —0.5000000 
—0.5000001 —0.5000001 —0.5000001 0.5000000 


A= 


75.8 Singular Value Decomposition 


Definitions: 


The singular value decomposition (SVD) of an m—by—n matrix A is 
A=UZXV! (A=UZXV*_ inthe complex case), (75.9) 


where U and V are orthogonal (unitary) and & is an m—by—n diagonal matrix with real diagonal elements, 
oj, such that 


01 2022...290, 0. 


min(m,n) = 


The o; are the singular values of A and the first min(m, n) columns of U and V are the left and right 
singular vectors of A. 


Backgrounds: 


The singular values o; and the corresponding left singular vectors u; and right singular vectors v; satisfy 
Av; =oju; and Atu; =ojvj (or A*uj=ojvj incomplex case), 


where uj; and v; are the jth columns of U and V, respectively. (See Chapter 17 and Chapter 45 for more 
information on singular value decompositions.) 
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Driver Routines: 


Two types of driver routines are provided in LAPACK. The simple driver xGESVD computes, all the 
singular values and (optionally) left and/or right singular vectors. The divide and conquer driver xGESDD 
has the same functionality as the simple driver except that it is much faster for larger matrices, but uses 
more workspace. 


Examples: 


Let us show how to use the simple driver SGESVD to compute the SVD (75.9). SGESVD first reduces A to 
a bidiagonal form, and then uses an implicit QR-type algorithm to compute singular values and optionally 
singular vectors. SGESVD has the following calling sequence: 


CALL SGESVD( JOBU, JOBVT, M, N, A, LDA, S, U, LDU, VT, LDVT, WORK, 
LWORK, INFO ) 


Input to SGESVD: 


JOBU: Specifies options for computing all or part of the left singular vectors U: 
= 'A',allM columns of U are returned in the array U: 
= 'S',the first min(M,N) columns of U are returned; 
= 'O', the first min(M,N) columns of U are overwritten on the array A; 
= 'N', no left singular vectors are computed. Note that JOBVT and JOBU cannot 
both be'0'. 
JOBVT: Specifies options for computing all or part of the right singular vectors V?: 
= 'A',all Nrows of V! are returned in the array VT; 
= 'S', the first min(M, N) rows of V! are returned; 
= 'O', the first min(M,N) rows of V7 are overwritten on the array A; 


= 'N', no right singular vectors are computed. 
M, N: The number of rows and columns of the matrix A.M, N > 0. 


A, LDA: The M—by-N matrix A and the leading dimension of the array A. LDA > 
max(1,M). 


LDU, LDVT: The leading dimension of the arrays U and VT. LDU, LDVT > 1; 
IfJOBU = 'S' or 'A',LDU>M. 
IfJOBVT = 'A',LDVT>N;IfJOBVT = 'S',LDVT >min(M,N). 


WORK, LWORK: The workspace array and its dimension. LWORK > max(3 min(M,N) + 
max(M, N), 5 min(M, N)). 


If LWORK = -1, then a workspace query is assumed; the routine only calculates the 
optimal size of the WORK array and returns this value as the first entry of the WORK 
array. 


Output from SGESVD: 


A: If JOBU = 'O', Ais overwritten with the first min(M, N) columns of U (the left 
singular vectors, stored columnwise); 
If JOBVT = 'O', A is overwritten with the first min(M,N) rows of V! (the right 


singular vectors, stored rowwise); 
S: Singular values of A, sorted so that S(i) > S(i + 1). 


U: IfJOBU = 'A',U contains M—by-M orthogonal matrix U. If JOBU = 'S',Ucon- 
tains the first min(M, N) columns of U. If JOBU = 'N' or 'O',Uisnot referenced. 
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vt: If JOBVT = 'A', VT contains right N-by—-N orthogonal matrix V". If JOBVT = 
'S', VT contains the first min(M, N) rows of V? (the right singular vectors stored 
rowwise). If JOBVT = 'N' or 'O', VT is not referenced. 


WORK: If INFO = 0,WORK(1) returns the optimal LWORK. 


INFO: =Oifsuccessful exit. If INFO = —j, the jth argument had an illegal value. If INFO > 
0, the QR-type algorithm (subroutine SBDSQR) did not converge. INFO specifies 
how many superdiagonals of an intermediate bidiagonal form B did not converge 
to zero. WORK(2:min(M, N)) contains the unconverged superdiagonal elements of an 
upper bidiagonal matrix B whose diagonal is in S (not necessarily sorted). B satisfies 
A = UBV’, 50 it has the same singular values as A, and singular vectors related by 
U and V?. 


Let us show the numerical results of SGESVD in computing the SVD by an 8-by-—5 matrix A as follows: 


22 10 2 3 
14 7 10 0 
-1 123 -1 -l1 


| 
ve 
| 
NO 
me 
Ow 
| 
NO 
NR RF SF BW ON 
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4 5 0 -2 


The exact singular singular values are ./ 1248, 20, / 384, 0,0. The rank of the matrix A is 3. Upon calling 
SGESVD with M = 8,N = 5, LWORK = 25, the computed singular values of A are returned in S: 


S = [35.3270454 20.0000038 19.5959187 0.0000007 0.0000004]. 


The columns in U contain the left singular vectors U of A: 


-7.0711e-001 1.5812e — 001 —1.7678e — 001 2.4818e — 001 —4.0289e — 001 —3.2305e — 001 —3.3272e — 001 —6.9129e — 002 
-5.3033e-001 1.58lle —001 3.5355e — 001 —6.24l6e —001 2.5591le — 001 —3.9178e — 002 3.0548e — 001 —1.3725e — 001 
-1.7678e-001 —7.9057e —001 1.7677e —001 3.0146e —001 1.9636e — 001 —3.1852e —001 2.3590e — 001 —1.6112e — 001 

U= 0 1.58lle—001 7.071le—001 2.9410e—001 3.1907e —001 4.7643e — 002 —5.2856e —001 7.1055e — 002 
-3.5355e-001 —1.58lle — 001 —1.0000e — 006 2.3966e — 001 —7.8607e — 002 8.7800e—001 1.0987e — 001 —5.8528e — 002 
-1.7678e-001 = 1.5812e — 001 —5.3033e —001 1.7018e —001 7.9071e — 001 —7.0484e — 003 —9.0913e — 002 —8.3220e — 004 

0 4.7434e—001 1.7678e —001 5.2915e — 001 —1.5210e — 002 —1.3789e —001 6.6193e —001 7.9763e — 002 

-1.7678e-001 —1.581le — 001 —1.0000e — 006 —7.1202e —002 1.3965e — 002 —2.0712e—002 4.9676e —002 9.6726e — 001 


The rows in VT contain the right singular vectors V" of A: 


—8.0064e — 001 4.8038e — 001 1.6013e — 001 0 —3.2026e — 001 
3.1623e —001 —6.3246e — 001 3.1622e — 001 6.3246e — 001 —1.8000e — 006 
vr = | —2.8867e — 001 —3.9000e — 006 8.6603e — 001 —2.8867e — 001 2.8868e — 001 


—4.0970e — 001 3.4253e —001 —1.2426e — 001 6.095le — 001 5.7260e — 001 
8.8224e — 002 5.0190e — 001 3.3003e — 001 —3.8100e — 001 6.9730e — 001 


75.9 Generalized Symmetric Definite Eigenproblems 


Definitions: 


The generalized symmetric definite eigenvalue problem (GSEP) is to find the eigenvalues, 1, and corre- 
sponding eigenvectors, x # 0, such that 


Ax =ABx (type 1) (75.10) 
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or 


ABx=dx (type 2) (75.11) 


or 


BAx=)dx (type 3) (75.12) 


where A and B are symmetric or Hermitian and B is positive definite. 


Backgrounds: 


For all these problems the eigenvalues A are real. The matrix Z of the computed eigenvectors satisfies 
Z* AZ = A (problem types 1 and 3) or Z~'AZ~* = I (problem type 2), where A is a diagonal matrix 
with the eigenvalues on the diagonal. Z also satisfies Z* BZ = I (problem types 1 and 2) or Z*B-1Z = I 
(problem type 3). These results are consequences of spectral theory for symmetric matrices. For example, 
the GSEP type 1 can be rearranged as 


B~? AB~?y=)hy, 


where y = B2x. 


Driver Routines: 


There are three types of driver routines for solving the GSEP, and each has variations that take advantage 
of the special structure or storage of the matrices A and B, as shown in the following table: 


Types of Matrix Routine Names 
(Storage Scheme) Simple Driver Divide-and-Conquer Expert Driver 
General dense xSYGV/xHEGV = xSYGVD/xHEGVD ~— xSYGVX/x HEGVX 
General dense 
(packed storage) | xSPGV/xHPGV = xSPGVD/xHPGVD — xSPGVX/xHPGVX 
Band matrix xSBGV/xHBGV = xSBBVD/xHBGVD ~~ xSBGVX/x HBGVX 


The simple driver computes all the eigenvalues and (optionally) the eigenvectors. The expert driver 
computes all or a selected subset of the eigenvalues and (optionally) eigenvectors. The divide-and-conquer 
driver solves the same problem as the simple driver. It is much faster than the simple driver, but uses more 
workspace. 


Examples: 


Let us show how to use the simple driver SSYGV to compute the GSEPs (75.10), (75.11), and (75.12). 
SSGYV first reduces each of these problems to a standard symmetric eigenvalue problem, using a Cholesky 
decomposition of B, and then computes eigenvalues and eigenvectors of the standard symmetric eigenvalue 
problem by an implicit QR-type algorithm. SSYGV has the following calling sequence: 

CALL SSYGV( ITYPE, JOBZ, UPLO,N, A, LDA, B, LDB, W, WORK, LWORK, INFO ) 

Input to SSYGV: 


ITYPE: Specifies the problem type to be solved: 


JOBZ: = 'N', compute eigenvalues only; 


= 'V', compute eigenvalues and eigenvectors. 


UPLO: = 'U',the upper triangles of A and B are stored; 


= 'L', the lower triangles of A and B are stored. 


N: The order of the matrices Aand B.N > 0. 


LAPACK 75-17 


A, LDA: The symmetric matrix A and the leading dimension of the array A. LDA > 
max(1,N). 


B: The symmetric positive definite matrix B and the leading dimension of the array B. 
LDB > max(1,N). 


WORK, LWORK: The workspace array and its length. LWORK > max(1,3 * N— 1). 


If LWORK = —1, then a workspace query is assumed; the routine only calculates the 
optimal size of the WORK array, and returns this value as the first entry of the WORK 
array. 


Output from SSYGV: 


A: Contains the normalized eigenvector matrix Z if requested. 


B: If INFO < N, the part of B containing the matrix is overwritten by the triangular 
factor U or L from the Cholesky factorization B = U'U or B= LL’. 
W: The eigenvalues in ascending order. 
WORK: If INFO = 0,WORK(1) returns the optimal LWORK. 
INFO: =0, then successful exit. If INFO = —j, then the jth argument had an illegal value. 
If INFO > 0, then SSYGV returned an error code: 
° INFO < N: if INFO = j, the algorithm failed to converge; 
° INFO > N: if INFO = N-+ j, for 1 < j <N, then the leading minor of order j of 
B is not positive definite. The factorization of B could not be completed and no 
eigenvalues or eigenvectors were computed. 


Let us show the use of SSYGV to solve the type 1 GSEP (75.10) for the following 5—by—5 matrices A and 
B: 


10 2 3 1 12 1 -l 2 1 

2. 12 I 2 1 14 1 -l 1 

Ar=>| By ol aE oD Sa and B= |!/-1 1 16 —-1l 1 
Les 2 1 9 2 -1 —-1 12 -!1 

1 1-1 1 #15 1 1 1 —-1 11 


Upon calling SSYGV with N = 5, LWORK = 3 * N— 1 = 14, A is overwritten by the eigenvector matrix Z: 


—0.1345906 0.0829197 —0.1917100 0.1420120 —0.0763867 
0.0612948 0.1531484 0.1589912 0.1424200 0.0170980 

Z= 0.1579026 —0.1186037 —0.0748390 0.1209976 —0.0666645 
—0.1094658 —0.1828130 0.1374690 0.1255310 0.0860480 
0.0414730 0.0035617 —0.0889779 0.0076922 0.2894334 


The corresponding eigenvalues are returned in W: 


w= [0.4327872 0.6636626 0.9438588 1.1092844 1.4923532] . 


75.10 Generalized Nonsymmetric Eigenproblems 


Definitions: 


The generalized nonsymmetric eigenvalue problem (GNEP) is to find the eigenvalues, 4, and corresponding 
(right) eigenvectors, x 4 0, such that 


Ax = X\Bx (75.13) 
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and optionally, the corresponding left eigenvectors y 4 0, such that 
y A= Ay*B, (75.14) 


where A and B are n—by—n matrices. In this section the terms right eigenvector and left eigenvector are used 
as just defined. 


Backgrounds: 


Sometimes an equivalent notation is used to refer to the GNEP of the pair (A, B). The GNEP can be solved 
via the generalized Schur decomposition of the pair (A, B), defined in the real case as 


A= OSZ', B= OTz, 


where Q and Z are orthogonal matrices, T is upper triangular, and S is an upper quasi-triangular matrix 
with 1-by-1 and 2-by-2 diagonal blocks, the 2-by-2 blocks corresponding to complex conjugate pairs of 
eigenvalues. In the complex case, the generalized Schur decomposition is 


A=QSZ*, B= QTZ"*, 


where Q and Z are unitary and S and T are both upper triangular. The columns of Q and Z are called left 
and right generalized Schur vectors and span pairs of deflating subspaces of A and B. Deflating subspaces 
are a generalization of invariant subspaces: For each k, 1 < k < n, the first k columns of Z span a right 
deflating subspace mapped by both A and B into a left deflating subspace spanned by the first k columns 
of Q. It is possible to order the generalized Schur form so that any desired subset of k eigenvalues occupies 
the k leading position on the diagonal of (S, T). (See Chapter 43 and Chapter 15 for more information on 
generalized eigenvalue problems.) 


Driver Routines: 


Both the simple and expert drivers are provided in LAPACK. The simple driver xGGEV computes all 
eigenvalues of the pair (A, B), and optionally the left and/or right eigenvectors. The expert driver xGGEVX 
performs the same task as the simple driver routines; in addition, it also balances the matrix pair to try to 
improve the conditioning of the eigenvalues and eigenvectors, and computes the condition numbers for 
the eigenvalues and/or left and right eigenvectors. 


Examples: 


Let us show how to use the simple driver SGGEV to solve the GNEPs (75.13) and (75.14). SGGEV first 
reduces the pair (A, B) to generalized upper Hessenberg form (H, R), where H is upper Hessenberg (zero 
below the first lower subdiagonal) and R is upper triangular. Then SGGEV computes the generalized Schur 
form (S, T) of the generalized upper Hessenberg form (H, R), using an QZ algorithm. The eigenvalues 
are computed from the diagonals of (S, T). Finally, SGGEV computes left and/or right eigenvectors if 
requested. SGGEV has the following calling sequence: 


CALL SGGEV( JOBVL, JOBVR, N, A, LDA, B, LDB, ALPHAR, ALPHAI, BETA, VL, 
LDVL, VR, LDVR, WORK, LWORK, INFO ) 


Input to SGGEV: 


JOBVL, JOBVR: = 'N',donot compute the left and/or right eigenvectors; 


= 'V', compute the left and/or right eigenvectors. 
N: The order of the matrices Aand B.N > 0. 


A, LDA: The matrix A and the leading dimension of the array A. LDA > max(1,N). 


B, LDB: The matrix B and the leading dimension of the array B. LDB > max(1,N). 
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LDVL, LDVR: Theleading dimensions of the eigenvector matrices VL and VR. LDVL, LDVR > 1. 


If eigenvectors are required, then LDVL, LDVR > N. 


WORK, LWORK: The workspace array and its length. LWORK > max(1,8 * N). For good 


performance, LWORK must generally be larger. 


If LWORK = —1, then a workspace query is assumed; the routine only calculates the 
optimal size of WORK, and returns this value in WORK (1) on return. 


Output from SGGEV: 


ALPHAR, ALPHAI, BETA: (ALPHAR(j)-+7i-ALPHAI(j))/BETA(j) for j = 1,2,...,N, 


VL: 


VR: 


are the generalized eigenvalues. If ALPHAI(]) is zero, then the jth eigenvalue is real; 
if positive, then the jth and (j + 1)-st eigenvalues are a complex conjugate pair, with 
ALPHAI(j + 1) negative. 


If JOBVL = 'V', the left eigenvectors y; are stored in the columns of VL, in the 
same order as their corresponding eigenvalues. If the jth eigenvalue is real, then 
yj = VL(:, j), the jth column of VL. Ifthe jth and (j +1)th eigenvalues forma complex 
conjugate pair, theny; = VL(:, j) ++7-VL(;, j +1) andyj4: = VL(, j) -i- VLG, j +1). 


If{JOBVR = 'V',theright eigenvectors x; are stored one after another in the columns 
of VR, in the same order as their eigenvalues. If the jth eigenvalue is real, then x; = 
VR(:, j), the jth column of VR. If the jth and (j + 1)th eigenvalues form a complex 
conjugate pair, thenx; = VR(:, j)+i-VR(:, j +1) andxj41 = VR(:, j)—i-VR(:, j +1). 


WORK: If INFO = 0, WORK(1) returns the optimal LWORK. 


INFO: INFO = 0 if successful exit. If INFO = —j, the jth argument had an illegal 


value. IFINFO = 1,...,N,then the QZ iteration failed. No eigenvectors have been 
calculated, but ALPHAR(j), ALPHAI(j), and BETA(j) should be correct for j = 
INFO + 1,...,N.If INFO = N+1, then other than QZ iteration failed in SHGEQZ. 
If INFO = N+2, then error return from STGEVC. 


Note that the quotients ALPHAR(j)/BETA(j) and ALPHAI(j)/BETA(j) may easily over- or underflow, and 
BETA(j) may even be zero. Thus, the user should avoid naively computing the ratio. However, ALPHAR 
and ALPHAT will be always less than and usually comparable to || A|| in magnitude, and BETA always less 
than and usually comparable to || B||. 

Let us demonstrate the use of SGGEV in solving the GNEP of the following 6—by—6 matrices A and B: 


The exact eigenvalues are 5 +i: , 
N = 6, LWORK = 48, on exit, arrays ALPHAR, ALPHAT, and BETA are 


50 —60 50 —27 6 6 16 #5 5 5 -6 5 
38 —28 27 —-17 5 5 5 16 5 5 -6 5 
27 —-17 27 —-17 5 = 5 ie GB Es 5 5 16 5 -6 5 
27 —28 38 —-17 5 5 5 5 5 16 -6 5 
27 —28 27 —-17 16 5 5 5 5 5 —-6 16 
27 —28 27 —-17 5 16 6 6 6 6 —5 6 


V3 1 » JZ 1 <3 4 
a i a ae a) 7? 


ALPHAR = [~25.7687130 6.5193458 5.8156629 5.8464251 5.5058141 11.2021322] , 


ree) 00, oo. Upon calling SGGEV with 


ALPHAT = [0.000000 11.2832556 —10.0653677 10.1340599 —9.5436525 0.000000] , 


BETA = {0.000000 13.0169611 11.6119413 11.7124090  11.0300474 0.000000] . 
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Therefore, there are two infinite eigenvalues corresponding to BETA(1) = BETA(6) = 0 and four finite 
eigenvalues 2; = (ALPHAR(j) + i - ALPHAI(j))/BETA(j) for j = 2,3,4,5. 


0.50083 + 7 - 0.86681 
0.50083 — 7 - 0.86681 
0.49917 +7 - 0.86524 
0.49917 —7 - 0.86524 


(ALPHAR(2 : 5) +i - ALPHAI(2 : 5))/BETA(2 : 5) = 


The left eigenvectors y; are stored in VL. Since ALPHAI(1) = ALPHAI(6) = 0, y; = VL(:,1) and ys = 
VL(:, 6). The second and third eigenvalues form a complex conjugate pair, the yp = VL(:,2) +7 - VL(:, 3) 
and y3 = VL(:,2) — 7 - VL(:,3). Similarly, yy = VL(:,4) +7- VL(:,5) and y; = VL(:,4) +7- VL(:,5). If we 
place all the left eigenvectors in a matrix Y, where Y = [y), y2, y3, yas Ys» Yo], we have 


—0.1666666 —0.2632965 +7 - 0.3214956 —0.2632965 — i - 0.3214956 —0.4613968 +7 -0.1902102 —0.4613968 —i-0.1902102  0.1666667 
—0.1666666 —0.2834885 —i -0.7165115 —0.2834885 +7-0.7165115 0.9231794 —i-0.0765849 —0.9231794 +i - 0.0765849 —0.1666667 

Y= —0.1666666  0.1623165 +7 -0.7526108 —0.1623165 — i - 0.7526108 —0.9240005 — i - 0.0759995 —0.9240005 +7 -0.0759995 —0.1666667 
—0.1666666  0.0396326 — i - 0.4130635 —-0.0396326 +7 - 0.4130635 —-0.4619284 +7 -0.1907958 —0.4619284 — 1 - 0.1907958 —0.1666666 
—0.1666671 —0.0605860 + i - 0.0184893 —0.0605860 — i - 0.0184893 —0.0000969 — 7 - 0.0761408 —0.0000969 +7 -0.0761408  0.1666666 
1.0000000 —0.0605855 +i - 0.0184900 —0.0605855 — i -0.0184900 —0.0000959 —i-0.0761405 —0.0000959 + 7 - 0.0761405 —1.0000000 


The right eigenvectors can be recovered from VR in a way similar to the left eigenvectors. If we place all 
the right eigenvectors in a matrix X, where X = [x), X),X3,X4, X5, X56], we have 


0.1666672 —0.2039835 — i - 0.5848466 —0.2039835 +1 -0.5848466 —0.5722237 — i - 0.0237538 = 0.5722237 +1 - 0.0237538 0.1666672 
0.1666664 —0.7090308 — i - 0.2908980 —0.7090308 + i -0.2908980 0.4485306 —7-0.5514694 —0.4485306 +17 - 0.5514694 0.1666664 

U= 0.1666666 —0.7071815 +7 - 0.2928185 —0.7071815 —i - 0.2928185 —0.0709520 — i - 0.7082051 —0.0709520 + i - 0.7082051 0.1666666 
~~ | 0.1666666 —0.2013957 +i - 0.5829236 —0.2013957 —i - 0.5829236 —0.4667411 — i - 0.3361499 —0.4667411 +i - 0.3361499 0.1666666 
1.0000000 —0.2023994 + i - 0.0000001 —0.2023994 —i-0.0000001  0.0536732 —i -0.1799536 —0.0536732 +i - 0.1799536 1.0000000 
0.1666666 —0.2023991 —i - 0.0000002 —0.2023991 +1 -0.0000002  0.0536734 —i-0.1799532 —0.0536734 +i - 0.1799532 0.1666664 


75.11 Generalized Singular Value Decomposition 


Definitions: 


The generalized (or quotient) singular value decomposition (GSVD or QSVD) of an m—by-n matrix A 
and a p—by-n matrix B is given by the pair of factorizations 


A=UZ,|0 R]Q™ and B=VE;[0 R] Q’. (75.15) 
The matrices in these factorizations have the following properties: 


° U is m—by—m, V is p—by—p, Q is n—by-n, and all three matrices are orthogonal. If A and B are 
complex, these matrices are unitary instead of orthogonal, and Q’ should be replaced by Q* in 
the pair of factorizations. 


* R is r—by-r, upper triangular and nonsingular. [0 R] is r—by—n (in other words, the 0 is an 
r—by—(n — r) zero matrix). The integer r is the rank of A : 
* ¥, is m-by-r and &) is p—by-r. Both are real, nonnegative, and diagonal, satisfying £7 E, + 


Dy L = I. Write L/D) = diag(a?,... ,a7) and UJ LX = diag(Ay,... , B?). The ratios a; /B; for 
j =1,2,...,1r are called the generalized singular values. 


, and XY have the following detailed structures, depending on whether m—r >Oorm-—r <0. 
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¢ In the first case, when m—r > 0, 


k 2 
k @ 
k/fI 0 il ae Ye 
y= £10 C and X)= : (75.16) 
p-£\0 0 
m—-k—-£\0 0 


Here k + € =r, and £ is the rank of B. C and S are diagonal matrices satisfying C* + S* = I, 
and S is nonsingular. Let c; and s; be the diagonal entries of C and S, respectively. Then we have 
a =) So = logy; = cj for j = 1,...,0, 8) = --- = Be = 0, and By; = s; for 
j = 1,...,€. Thus, the first k generalized singular values a /f,... ,a@%/Bx are infinite and the 
remaining € generalized singular values are finite. 

* In the second case, when m—r < 0, 


k m—-k kt+t€—-m 
k m—-k k+l—-m 
Pie he ‘ ‘ m—k [0 S 0 
y= and 2,=k+l—m] 0 0 I 
m—k \0 C 0 
PENG x0 0 
(75.17) 


Again, k + £ =r, and £ is the rank of B. C and S are diagonal matrices satisfying C? + S? = I, 
and S is nonsingular. Let c; and s; be the diagonal entries of C and S, respectively. Then we 


have og) = --- = ay = 1, oy; = cj for j = 1,...,m—k, Amy, = ++: = a, = 0, and 
Bi =--- = Be = 0, Bey j = sj for j = 1,...,m—k, Bngi = --- = B, = 1. Thus, the first k 
generalized singular values a) /f1,... ,@%/B, are infinite, and the remaining ¢ generalized singular 


values are finite. 


Backgrounds: 

Here are some important special cases of the QSVD. First, when B is square and nonsingular, then r = n 
and the QSVD of A and B is equivalent to the SVD of AB~!, where the singular values of AB~! are equal 
to the generalized singular values of A and B: 


AB™! = (UX,RQ")(VX2RQ*)1 = U(X, =Uz"')V" . 


T 

Second, if the columns of a? B"] are orthonormal, then r = n, R = I, and the QSVD of A and B 
T 

is equivalent to the CS (Cosine-Sine) decomposition of [at B J : 

A x1) yr 

: Jo. 


U 0 
0 V 


X2 


Third, the generalized eigenvalues and eigenvectors of the pencil A’ A — 1B’ B can be expressed in terms 
of the QSVD of A and B, namely, 


0 0 0 
XTATAX = d X’B'BX= ; 
F si, 7 aa 
I 0 : 
where X = Q 0 Rol: Therefore, the columns of X are the eigenvectors of A’ A — AB" B, and the 


“nontrivial” eigenvalues are the squares of the generalized singular values. “Trivial” eigenvalues are those 
corresponding to the leading n — r columns of X, which span the common null space of A? A and B’ B. 
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The “trivial eigenvalues” are not well defined.! (See Chapter 15 for more information on generalized 


singular value problems.) 


Driver Routines: 
The driver routine xGGSVD computes the GSVD (75.15) of A and B. 


Examples: 

Let us show how to use the driver routine SGGSVD to compute the QSVD (75.15). SGGSVD first reduces 
the matrices A and B to a pair of triangular matrices, and then use a Jacobi-like method to compute the 
QSVD of the triangular pair. SGGSVD has the following calling sequence: 


CALL SGGSVD( JOBU, JOBV, JOBQ, M, N, P, K, L, A, LDA, B, LDB, ALPHA, 


BETA, U, LDU, V, LDV, Q, LDQ, WORK, IWORK, INFO ) 


Input to SGGSVD: 


JOBU, JOBV, JOBQ: ,= 'U', orthogonal matrices U, V and Q are computed; 


= 'N', these orthogonal matrices are not computed. 
M, N, P: The number of rows or columns of the matrices A and B as defined in (15) 


A, LDA: TheM—by—N matrix A and theleading dimension ofthe array A.LDA > max(1,M). 


B, LDB: The P—by—N matrix B and the leading dimension ofthe array B.LDB > max(1,P). 


iDU, LDV, LDQ: The leading dimension of the arrays U, V, and Q if the orthogonal 
matrices U, V, and Q are computed, LDU > max(1,M), LDV > max(1,P), LDQ > 
max(1,N). 


WORK: The real workspace array, dimension max(3N, M,P) +N. 


IWORK: The integer workspace array, dimension N. 


Output from SGGSVD: 


K, L: The dimension of the subblocks described in the definition of GSVD. K + L is the 


T 
effective numerical rank of the matrix [at B " : 


A: The entire triangular matrix R is stored in A(1:K+L,N-K-L+1:N) ifm—r > 0. 
Otherwise, the subblock R(1 : m,1:k+4€) of RarestoredinA(1:M,N-K-L+1:N). 


B: The subblock R(m+1:k+,m+1:k-+ £) of R are stored in B(M-K+1:L,N+M- 
K-L+1:N) ifm—r <0. 


ALPHA, BETA: The generalized singular value pairs; 
ALPHA(1:K) = landBETA(1:K) = 0. 


¢ IfM-K-L > 0, then ALPHA(K+1:K+L)=C andBETA(K+1:K+L)=S. 
¢ IfM-K-L < 0, then 
ALPHA(K+1:M)=C and ALPHA(M+1:K+L)=0, 
BETA(K+1:M)=S and BETA(M+1:K+L)=1; 


And ALPHA (K+L+1:N) = 0,BETA(K+L+1:N) = 0. 


U, V, Q: Contains computed orthogonal matrices U, V, and Q if requested. 


‘If we tried to compute the trivial eigenvalues in the same way as the nontrivial ones, that is by taking ratios of the 
leading n — r diagonal entries of X' A? AX and X7 B™ BX, we would get 0/0. 
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INFO: INFO = 0 if successful exit. If INFO = —j, then the jth argument had an illegal 
value. If INFO = 1, the Jacobi-type procedure failed to converge. 


Let us demonstrate the use of SGGSVD in computing the QSVD of the following 6—by—5 matrices A 
and B: 


a ok ae 1) aso, GS al 
03-2 0 2 0 3 °0 @ 0 
ie ey ae fie Oe 

Bea a ae igs ig. |) Re ee oy agg 
ae a a ae a ae ae ee 
OS te -a ae ae ee es 


Upon calling SGGSVD with M = 6,P = 6,N = 5, LWORK = 20, we have K = 2 and L = 2. The QSVD 
(75.15) of A and B falls in the first case (75.16) sinceM—K—L=6—2—2=2 > 0. The arrays ALPHA 
and BETA are 


ALPHA = [1.000000 1.0000000 0.1537885 0.5788464 0.000000] , 


BETA = [0.000000 0.0000000 0.9881038 0.8154366 0.000000] . 


Hence, %; and X have the structure as described in (75.16), namely, 


1 0 0 0 0 0 0.9881038 0 
0 1 0 0 0 0 0 0.8154366 
0 0 0.1537885 0 0 0 0 0 
= 19 0 0 05788464, M4 = Iq g 0 0 
0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 
The first two generalized singular values are infinite, a,/B, = a@2/B, = oo, and the remaining two 


generalized singular values are finite, «3/f3 = 0.15564 and a4/B4 = 0.70986. 
Furthermore, the array A(1:4,2:5) contains the 4-by—4 upper triangular matrix R as defined in 
(75.15): 


3.6016991 —1.7135643 —0.2843603 1.8104467 
0 —2.6087811 —4.2943931 5.1107349 
0 0 6.9692163 3.5063875 
0 0 0 7.3144341 


The orthogonal matrices U, V, and Q are returned in the arrays U, V, and Q, respectively: 


| —0.6770154 0.4872811 0.4034495 0.2450049 0.2151961 0.1873468) 
—0.0947438 —0.5723576 0.4163284 0.1218751 0.0785425 —0.6848933 
0.2098812 0.0670342 0.2612190 0.7393155 0.5670457 0.1228532 


he 0.6974092 —0.5903998 —0.3678919 0.0010751 —0.0196356 0.1712235| ° 
0.0000000 0.0000001 —0.0735656 —0.6152450 0.7822418 —0.0644937 
L—0.0473719 —0.2861788 0.6744684 —0.0019711 0.1170180 0.6687696] 
f—0.3017521 —0.2581125 0.9018297 0.0002676 0.1695592 0.0166328] 
0.4354534 —0.2679386 0.1028928 0.0704557 0.2595005 —0.8097517 
Ve —0.3017520 0.2581124 0.1784097 0.8828155 0.0002829 0.1764375 


0.2903022 0.1786257 0.1298870 0.0008522 0.9259184 0.0980879 | ° 
—0.6035041 —0.5162248 —0.3568195 0.4625078 —0.0224125 —0.1660080 
L 0.4240036 0.7046767 0.0097810 0.0419325 0.2146671 0.5250862] 


| —0.7071068 0.2073452 0.5604916 0.0112638 0.3777966 
0.0000000 0.0000000 0.0000000 0.9995558 —0.0298012 
Q= 0.0000001 0.5853096 0.2932303 —0.0225276 —0.7555932] . 

0.7071067 0.2073452 0.5604916 0.0112638 0.3777965 
L—0.0000001 —0.7559289 0.5345224 —0.0112638 Sp sprises| 
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ARPACK is a library of Fortran77 subroutines designed to compute a selected subset of the eigenvalues of 
a large matrix. It is based upon a limited storage scheme called the implicitly restarted Arnoldi method 
(IRAM) [Sor92]. This software can solve largescale non-Hermitian or Hermitian (standard and gener- 
alized) eigenvalue problems. The IRA method is described in Chapter 44, Implicitly Restarted Arnoldi 
Method. 

This chapter describes the design and performance features of the eigenvalue software ARPACK and 
gives a brief discussion of usage. More detailed descriptions are available in the papers [Sor92] and [Sor02] 
and in the ARPACK Users’ Guide [LSY98]. 

The design goals were robustness, efficiency, and portability. Two very important principles that have 
helped to achieve these goals are modularity and independence from specific vendor supplied communi- 
cation and performance libraries. 

In this chapter, multiplication of a vector x by a scalar A is denoted by xA so that the eigenvector— 
eigenvalue relation is Ax = xd. This convention provides for direct generalizations to the more general 
invariant subspace relations AX = X H, where X is ann x k matrix and H isak x k matrix with k <n. 


76.1 The ARPACK Software 


The ARPACK software has been used on a wide range of applications. P-ARPACK is a parallel extension to 
the ARPACK library and is targeted for distributed memory message passing systems. Both packages are 
freely available and can be downloaded at http: //www.caam.rice.edu/software/ARPACK/. 
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Features (of ARPACK and P_ARPACK): 


1. 
2. 


A reverse communication interface. 
Computes k eigenvalues that satisfy a user-specified criterion, such as largest real part, largest 
absolute value, etc. 


. A fixed predetermined storage requirement of n - O(k) + O(k?) words. 


Driver routines are included as templates for implementing various spectral transformations to 
enhance convergence and to solve the generalized eigenvalue problem, or the SVD problem. 
Special consideration is given to the generalized problem Ax = Mxd for singular or ill-conditioned 
symmetric positive semidefinite M. 


. A Schur basis of dimension k that is numerically orthogonal to working precision is always com- 


puted. These are also eigenvectors in the Hermitian case. In the non-Hermitian case eigenvectors 
are available on request. Eigenvalues are computed to a user specified accuracy. 


76.2 Reverse Communication 


Reverse communication is a means to overcome certain restrictions in the Fortran language; with reverse 
communication, control is returned to the calling program when interaction with the matrix is required. 
This is a convenient interface for experienced users. However, it may be more challenging for inexperienced 
users. It has proven to be extremely useful for interfacing with large application codes. 

This interface avoids having to express a matrix-vector product through a subroutine with a fixed 
calling sequence or to provide a sparse matrix with a specific data structure. The user is free to choose any 
convenient data structure for the matrix representation. 


Examples: 


1. 


A typical use of this interface is illustrated as follows: 


10 continue 
call snaupd (ido, bmat, n, which,..., workd,..., info) 
if (ido .eq. newprod) then 
call matvec ('A', n, workd(ipntr(1)), workd(ipntr (2) )) 
else 
return 
endif 
go to 10 


% 


This shows a code segment of the routine the user must write to set up the reverse communication 
call to the top level ARPACK routine snaupd to solve a nonsymmetric eigenvalue problem in 
single precision. With reverse communication, control is returned to the calling program when 
interaction with the matrix A is required. The action requested of the calling program is specified 
by the reverse communication parameter ido, which is set in the call to snaupd. In this case, 
there are two possible requests indicated by ido. One action is to multiply the vector held in 
the array workd beginning at location ipntr(1) by A and then place the result in the array 
workd beginning at location ipntr (2). The other action is to halt the iteration due to successful 
convergence or due to an error. 

When the parameter ido indicates a new matrix vector product is required, a call is made to a 
subroutine matvec in this example. However, it is only necessary to supply the action of the matrix 
on the specified vector and put the result in the designated location. No specified data structure is 
imposed on A and if a subroutine is used, no particular calling sequence is specified. Because of 
this, reverse communication is very flexible and even provides a convenient way to use ARPACK 
interfaced with code written in another language, such as C or C++. 
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FIGURE 76.1 The ARPACK directory structure. 


76.3 Directory Structure and Contents 


Once the ARPACK software has been downloaded and unbundled, a directory structure will have been 
created as pictured in Figure 76.1. 


Subdirectories: 


1. 


The ARMAKES subdirectory contains sample files with machine specific information needed 
during the building of the ARPACK library. 

The BLAS and LAPACK subdirectories contain the necessary codes from those libraries. 

The DOCUMENTS subdirectory contains files that have example templates showing how to invoke 
the different computational modes offered by ARPACK. 

Example driver programs illustrating all of the computational modes, data types, and precisions 
may be found in the EXAMPLES directory. 


. Programs for banded, complex, nonsymmetric, symmetric eigenvalue problems, and singular value 


decomposition may be found in the directories BAND, COMPLEX, NONSYM, SYM, SVD. 

The README files in each subdirectory provide further information. 

The SRC subdirectory contains all the ARPACK source codes. 

The UTIL subdirectory contains the various utility routines needed for printing results and timing 
the execution of the ARPACK subroutines. 


76.4 Naming Conventions, Precisions, and Types 


1 


2. 


. ARPACK has two interface routines that must be invoked by the user. They are __aupd that imple- 


ments the IRAM and __eupd to post process the results of __aupd. 

The user may request an orthogonal basis for a selected invariant subspace or eigenvectors cor- 
responding to selected eigenvalues with --eupd. If a spectral transformation is used, --eupd 
automatically transforms the computed eigenvalues of the shift-invert operator to those of the 
original problem. 


. Both __aupd and __eupd are available for several combinations of problem type (symmetric 


and nonsymmetric), data type (real, complex), and precision (single, double). The first letter 
(s,d,c,z) denotes precision and data type. The second letter denotes whether the problem is 
symmetric (s) or nonsymmetric (n). 
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4, Thus, dnaupd is the routine to use if the problem is a double precision nonsymmetric (standard 
or generalized) problem and dneupd is the post-processing routine to use in conjunction with 
dnaupd to recover eigenvalues and eigenvectors of the original problem upon convergence. For 
complex matrices, one should use naupd and _neupd with the first letter either c or z regardless 
of whether the problem is Hermitian or non-Hermitian. 


76.5 Getting Started 


Perhaps the easiest way to rapidly become acquainted with the possible uses of ARPACK is to run the 
example driver routines that have been supplied for each of the computational modes. These may be used 
as templates and adapted to solve specific problems. To get started, it is recommended that the user execute 
driver routines from the SIMPLE subdirectory. 

The dssimp driver implements the reverse communication interface to the routine dsaupd that will 
compute a few eigenvalues and eigenvectors of a symmetric matrix. It illustrates the simplest case and 
has exactly the same structure as shown previously except that the top level routine is dsaupd instead of 
snaupd. The full call issued by dssimp is as follows. 


call dsaupd ( ido, bmat, n, which, nev, tol, resid, 
& nev, v, ldv, iparam, ipntr, workd, 
& workl, lworkl, info ) 


This dssimp driver is intended to serve as template to enable a user to create a program to use dsaupd 
on a specific problem in the simplest computational mode. All of the driver programs in the various 
EXAMPLES subdirectories are intended to be used as templates. They all follow the same principle, but the 
usage is slightly more complicated. 

The only thing that must be supplied in order to use this routine on your problem is to change the array 


dimensions and to supply a means to compute the matrix-vector product 
w< Av 


on request from dsaupd. The selection of which eigenvalues to compute may be altered by changing the 
parameter which. 

Once usage of dsaupd in the simplest mode is understood, it will be easier to explore the other available 
options such as solving generalized eigenvalue problems using a shift-invert computational mode. 

If the computation is successful, dsaupd indicates that convergence has taken place through the pa- 
rameter ido. Then various steps may be taken to recover the results in a useful form. This is done through 
the subroutine dseupd as illustrated below. 


call dseupd(rvec, howmny, select, d, v, ldv, sigma, bmat, 
& n, which, nev, tol, resid, nev, v, ldv, 
& iparam, ipntr, workd, workl, lworkl, ierr) 


Eigenvalues are returned in the first column of the two-dimensional array d and the corresponding 
eigenvectors are returned in the first NCONV (=IPARAM(5)) columns of the two-dimensional array v 
if requested. Otherwise, an orthogonal basis for the invariant subspace corresponding to the eigenvalues 
in dis returned in v. 

The input parameters that must be specified are 


* The logical variable rvec = . true. if eigenvectors are requested, 
. false. if only eigenvalues are desired. 
* The character*1 parameter howmny that specifies how many eigenvectors are desired. 
howmny = 'A': compute nev eigenvectors; 
howmny = 'S': compute some of the eigenvectors, 
specified by the logical array select. 
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* sigma should contain the value of the shift used ifiparam(7) = 3,4,5. Itis not referenced if 
iparam(7) = 1 or 2. 


When requested, the eigenvectors returned by dseupd are normalized to have unit length with respect 
to the M semi-inner product that was used. Thus, if M = I, they will have unit length in the standard 
2-norm. In general, a computed eigenvector x will satisfy 1 = x7 Mx. 


76.6 Setting Up the Problem 


To set up the problem, the user needs to specify the number of eigenvalues to compute which eigenvalues are 
of interest, the number of basis vectors to use, and whether or not the problem is standard or generalized. 
These items are controlled by the following parameters. 

Parameters for the top-level ARPACK routines: 


ido — Reverse communication flag. 

nev — The number of requested eigenvalues to compute. 

nev — The number of Arnoldi basis vectors to use through the course of the computation. 

bmat — Indicates whether the problem is standard bmat = 'I' orgeneralized(bmat = 'G'). 
which — Specifies which eigenvalues of A are to be computed. 

tol — Specifies the relative accuracy to which eigenvalues are to be computed. 

iparam — Specifies the computational mode, number of IRAM iterations, the implicit shift 
strategy, and outputs various informational parameters upon completion of IRAM. 


The value of ncv must be at least nev + 1. The options available for which include 'LA' and 'SA' for 
the algebraically largest and smallest eigenvalues, 'LM' and 'SM' for the eigenvalues of largest or smallest 
magnitude, and 'BE' for the simultaneous computation of the eigenvalues at both ends of the spectrum. 
Fora given problem, some of these options may converge more rapidly than others due to the approximation 
properties of the IRAM as well as the distribution of the eigenvalues of A. Convergence behavior can be 
quite different for various settings of the which parameter. For example, if the matrix is indefinite then 
setting which = 'SM' will require interior eigenvalues to be computed and the Arnoldi/Lanczos process 
may require many steps before these are resolved. 

For a given nev, the computational work required is proportional to n - ncv? FLOPS. Setting nev and 
ncv for optimal performance is very much problem dependent. If possible, it is best to avoid setting nev 
in a way that will split clusters of eigenvalues. For example, if the the five smallest eigenvalues are positive 
and on the order of 10~4 and the sixth smallest eigenvalue is on the order of 10~', then it is probably better 
to ask for nev = 5 than for nev = 3, even if the three smallest are the only ones of interest. 

Setting the optimal value of ncv relative to nev is not completely understood. As with the choice of 
which, it depends upon the underlying approximation properties of the IRAM as well as the distribution 
of the eigenvalues of A. As a rule of thumb, ncv > 2 - nev is reasonable. There are tradeoffs due to the 
cost of the user supplied matrix-vector products and the cost of the implicit restart mechanism and the 
cost of maintaining the orthogonality of the Arnoldi vectors. If the user supplied matrix-vector product is 
relatively cheap, then a smaller value of ncv may lead to more user matrix-vector products, but an overall 
decrease in computation time. 


Storage Declarations: 


The program is set up so that the setting of the three parameters maxn, maxnev, maxncv will auto- 
matically declare all of the work space needed to run dsaupd on a given problem. 

The declarations allow a problem size of N < maxn, computation of nev < maxnev eigenvalues, 
and using at most ncv < maxncv Arnoldi basis vectors during the IRAM. The user may override the 
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Double precision 
& v(ldv,maxncv), workl (maxncv* (maxncv+8)), 
& workd(3*maxn), d(maxncv,2), resid(maxn) , 


FIGURE 76.2 Storage declarations needed for ARPACK subroutine dsaupd. 


default settings used for the example problem by modifying maxn, maxnev, and maxncv in the following 
parameter statement in the dssimp code. 


integer maxn, maxnev, maxncv, ldv 
parameter (maxn=256, maxnev=10, maxncev=25, ldv=maxn) 


These parameters are used in the code segment listed in Figure 76.2 for declaring all of the output and 
work arrays needed by the ARPACK subroutines dsaupd and dseupd. These will set the storage values 
in ARPACK arrays. 


Stopping Criterion: 


The stopping criterion is determined by the user through specification of the parameter tol. The default 
value for tol is machine precision €,y. There are several things to consider when setting this parameter. 
In absence of all other considerations, one should expect a computed eigenvalue 1. to roughly satisfy 


|Ac — At| < toll Allo, 


where A; is the eigenvalue of A nearest to 4,. Typically, decreasing the value of tol will increase the 
work required to satisfy the stopping criterion. However, setting tol too large may cause eigenvalues to 
be missed when they are multiple or very tightly clustered. Typically, a fairly small setting of tol and 
a reasonably large setting of ncv is required to avoid missing multiple eigenvalues. However, some care 
must be taken. It is possible to set tol so small that convergence never occurs. There may be additional 
complications when the matrix A is nonnormal or when the eigenvalues of interest are clustered near the 
origin. 


Initial Parameter Settings: 


The reverse communication flag is denoted by ido. This parameter must be initially set to 0 to signify 
the first call to dsaupd. Various algorithmic modes may be selected through the settings of the entries in 
the integer array iparam. The most important of these is the value of iparam(7), which specifies the 
computational mode to use. 


Setting the Starting Vector: 


The parameter info should be set to 0 on the initial call to dsaupd unless the user wants to supply the 
starting vector that initializes the IRAM. Normally, this default is a reasonable choice. However, if this 
eigenvalue calculation is one of a sequence of closely related problems, then convergence may be accelerated 
ifa suitable starting vector is specified. Typical choices in this situation might be to use the final value of 
the starting vector from the previous eigenvalue calculation (that vector will already be in the first column 
of V) or to construct a starting vector by taking a linear combination of the computed eigenvectors from 
the previously converged eigenvalue calculation. If the starting vector is to be supplied, then it should be 
placed in the array residand info shouldbe set to 1 on entry to dsaupd. On completion, the parameter 
info may contain the value 0 indicating the iteration was successful or it may contain a nonzero value 
indicating an error or a warning condition. The meaning of a nonzero value returned in info may be 
found in the header comments of the subroutine dsaupd. 
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Trace Debugging Capability: 


ARPACK provides a means to trace the progress of the computation as it proceeds. Various levels of output 
may be specified from no output (level = 0 ) to voluminous (level = 3). A detailed description 
of trace debugging may be found in [LSY98]. 


76.7 General Use of ARPACK 


The Shift and Invert Spectral Transformation Mode: 


The most general problem that may be solved with ARPACK is to compute a few selected eigenvalues and 
corresponding eigenvectors for 


Ax= Mx), 


where A and M are real or complex n x n matrices. 
The shift and invert spectral transformation is used to enhance convergence to a desired portion of the 
spectrum. If (x, A) is an eigen-pair for (A, M) ando # A, then 
+1 1 
(A—oM)'Mx=xv where v= , 


X-o 


where we are requiring that A — o M is nonsingular. Here it is possible for A or M to be singular, but they 
cannot have a nonzero null vector in common. This transformation is effective for finding eigenvalues 
near o since the nev eigenvalues v; of C = (A—o M)~' M thatare largest in magnitude correspond to the 
nev eigenvalues A; of the original problem that are nearest to the shift o in absolute value. As discussed in 
Chapter 44, these transformed eigenvalues of largest magnitude are precisely the eigenvalues that are easy 
to compute with a Krylov method. Once they are found, they may be transformed back to eigenvalues of 
the original problem. 


M Is Hermitian Positive Definite: 


If M is Hermitian positive definite and well conditioned (|| M|| - || M~'|| is of modest size), then computing 
the Cholesky factorization M = L L* and converting Ax = Mxa into 


(L“'AL~™*)y=ya, where L*x=y 


provides a transformation to a standard eigenvalue problem. In this case, a request for a matrix vector 
product would be satisfied with the following three steps: 


1. Solve L*z = v for z. 
2. Matrix-vector multiply z < Az. 
3. Solve Lw = z for w. 


Upon convergence, a computed eigenvector y for (L~' AL ~*) is converted to an eigenvector x of the original 
problem by solving the the triangular system L*x = y. This transformation is most appropriate when 
A is Hermitian, M is Hermitian positive definite, and extremal eigenvalues are sought. This is because 
L~' AL~* will be Hermitian when A is the same. 

If A is Hermitian positive definite and the smallest eigenvalues are sought, then it would be best to 
reverse the roles of A and M in the above description and ask for the largest algebraic eigenvalues or 
those of largest magnitude. Upon convergence, a computed eigenvalue 4 would then be converted to an 
eigenvalue of the original problem by the relation A <— 1/A. 


M Is NOT Hermitian Positive Semidefinite: 


If neither A nor M is Hermitian positive semidefinite, then a direct transformation to standard form is 
required. One simple way to obtain a direct transformation to a standard eigenvalue problem Cx = xa 
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is to multiply on the left by M~', which results in C = M~'A. Of course, one should not perform this 
transformation explicitly since it will most likely convert a sparse problem into a dense one. If possible, 
one should obtain a direct factorization of M and when a matrix-vector product involving C is called for, 
it may be accomplished with the following two steps: 


1. Matrix-vector multiply z <— Av. 
2. Solve: Mw = z. 


Several problem dependent issues may modify this strategy. If M is singular or if one is interested in 
eigenvalues near a point o, then a user may choose to work with C = (A — 0 M)~'M but without using 
the M-inner products discussed previously. In this case the user will have to transform the converged 
eigenvalues v; of C to eigenvalues A; of the original problem. 


76.8 Using the Computational Modes 


An extensive set of computational modes has been constructed to implement all of the shift-invert options 
mentioned previously. The problem set up is similar for all of the available computational modes. A 
detailed description of the reverse communication loop for the various modes (Shift-Invert for a Real 
Nonsymmetric Generalized Problem) is available in the users’ guide [LSY98]. To use any of the modes 
listed below, the user is strongly urged to modify one of the driver routine templates that reside in the 
EXAMPLES directory. 


When to Use a Spectral Transformation: 


The first thing to decide is whether the problem will require a spectral transformation. If the problem is 
generalized (M # I), then a spectral transformation will be required. Such a transformation will most 
likely be needed for a standard problem if the desired eigenvalues are in the interior of the spectrum or if 
they are clustered at the desired part of the spectrum. Once this decision has been made and OP has been 
specified, an efficient means to implement the action of the operator OP on a vector must be devised. The 
expense of applying OP to a vector will of course have direct impact on performance. 

Shift-invert spectral transformations may be implemented with or without the use of a weighted 
M-inner product. The relation between the eigenvalues of OP and the eigenvalues of the original problem 
must also be understood in order to make the appropriate specification of which in order to recover eigen- 
values of interest for the original problem. The user must specify the number of eigenvalues to compute, 
which eigenvalues are of interest, the number of basis vectors to use, and whether or not the problem is 
standard or generalized. 


Computational Modes for Real Nonsymmetric Problems: 


The following subroutines are used to solve nonsymmetric generalized eigenvalue problems in real 
arithmetic. These routines are appropriate when A is a general nonsymmetric matrix and M is sym- 
metric and positive semidefinite. The reverse communication interface routine for the nonsymmetric 
double precision eigenvalue problem is dnaupd. The routine is called as shown below. The specification 
of which nev eigenvalues is controlled by the character*2 argument which. The most commonly 
used options are listed below. There are templates available as indicated for each of these. 


call dnaupd (ido, bmat, n, which, nev, tol, resid, ncv, v, 
& ldv, iparam, ipntr, workd, workl, lworkl, info) 


There are three different shift-invert modes for nonsymmetric eigenvalue problems. These modes are 
specified by setting the parameter entry iparam(7) = mode wheremode = 1,2,3, or 4. 

In the following list, the specification of OP and B are given for the various modes. Also, the iparam (7) 
and bmat settings are listed along with the name of the sample driver for the given mode. Sample drivers 
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for the following modes may be found in the EXAMPLES /NONSYM subdirectory. 
1. Regular mode (iparam(7) = 1, bmat = 'I' ) 
Use driver dndrv1. 
(a) Solve Ax = xd in regular mode. 
(b) OP = Aand B=I. 
2. Shift-invert mode (iparam(7) = 3, bmat = 'I'). 
Use driver dndrv2 with sigma a real shift. 
(a) Solve Ax = xA in shift-invert mode. 
(b) OP = (A—oI)7! and B =I. 
3. Regular inverse mode (iparam(7) = 2, bmat = 'G') 
Use driver dndrv3. 
(a) Solve Ax = Mx in regular inverse mode. 
(b) OP = M~'Aand B= M. 
4. Shift-invert mode (iparam(7) = 3, bmat = 'G'). 


Use driver dndrv4 with sigma a real shift. 
(a) Solve Ax = Mxa in shift-invert mode. 
(b) OP = (A—oM)-!Mand B= M. 


76.9 MATLAB’S® EIGS 


MATLAB has adopted ARPACK for the computation of a selected subset of the eigenvalues ofa large (sparse) 
matrix. The MATLAB function for this is called eigs and it isa MATLAB driver to a mex-file compilation of 
ARPACK. In fact, a user can directly reference this mex-file as arpackc. Exactly the same sort of interfaces 


discussed above can be written in MATLAB to drive arpa 


However, it is far more convenient just to use the provided eigs function. The command D 
returns a vector of the 6 eigenvalues of A of largest magnitude, while [V,D] 


eigenvectors in V and the corresponding (6 largest magnit 
The command eigs (A,M) solves the generalized eig 
be symmetric (or Hermitian) positive definite and the sa 


cke. 


eigs (A) 
returns 


eigs (A) 
ude) eigenvalues on the diagonal of the matrix D. 
envalue problem A*V = M*V*D. Here M must 
me size as A. If M is nonsymmetric, this can also 


be handled directly, but the user will have to write and pass a function that will compute the actionw <- 
inv(A - sigma*M)* M*v_ as needed. Of course, a sparse direct factorization of (A - sigma*M) 


should be computed at the outset and reused to accompl 


ish this action. 


Other capabilities are easily utilized through eigs. The commands eigs (A,k) and eigs (A,M,k) 


return the k largest magnitude eigenvalues. 
The commands eigs(A,k,sigma) and eigs(A, 


M,k,sigma) return k eigenvalues based on 


sigma, where sigma is either a scalar or one of the following strings: 'LM' or 'SM' - Largest or Smallest 


Magnitude 


For real symmetric problems, 


'LA' or 'SA' - Largest or Small 
'BE' - Both Ends, 
is odd 


may also be: 
LR' or 'SR' 
Br SL" 
part 


or 


SIGMA may also be: 


lest Algebraic 


one more from high end if K 


For nonsymmetric and complex problems, SIGMA 


Largest or Smallest Real part 


Largest or Smallest Imaginary 
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If sigma is a scalar including 0, eigs finds the eigenvalues in the complex plane closest to sigma . 
The required sparse matrix factorizations are done automatically when a scalar is passed. 

More control and flexibility can be obtained by using the commands eigs (A,M, sigma, opts) and 
eigs(A,M,k,sigma,opts) whereopts isastruct. The fields of the struct opts will specify 
the following options: 


opts.issym: symmetry of A or A-SIGMA*B 
represented by AFUN [{0} | 1] 


opts.isreal: complexity of A or A-SIGMA*B 
represented by AFUN [0 | {1} 


opts.tol: convergence: Ritz estimate 
residual <= tol*NORM(A) [scalar | {eps}] 


opts.maxit: maximum number of 
iterations [integer | {300}] 


opts.p: number of Lanczos vectors: 
K+1<p<=N [integer {2K} ] 


opts.v0: starting vector 
[N-by-1 vector {randomly generated by ARPACK}] 


opts.disp: diagnostic information display 
level [0 | {1} 2 


opts.cholB: B is actually its Cholesky 
factor CHOL(B) [{0} | 1] 


opts.permB: sparse B is actually 
CHOL(B(permB,permB)) [permB | {1:N}] 


opts(AFUN,n) accepts the function AFUN 
instead of a matrix 


In the above list, the items in square brackets denote the possible settings and the item given in curly 
brackets {-} is the default setting. For example, to set the convergence tolerance to 10~%, one should give 
the command opts.tol = .001 . 

The function passing mechanism can be used to define a customized spectral transformation as in the 
example with a nonsymmetric M. Arguments may be passed to AFUN(X, P1,P2,...) by invoking the 
command eigs (AFUN,n,k,sigma,opts,P1,P2,...). 

Thus, all of the functionality of ARPACK is available through eigs and, in many cases, it is much easier 
to use. Moreover, the sparse matrix direct factorizations and reorderings available in the spar fun suite 
are easily used to implement various desirable spectral transformations. It is highly advisable to run sample 
problems with the characteristics of a desired very large problem in order to get an idea of the best spectral 
transformation to use and to get an indication of the expected convergence behavior. 
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Table 77.1 to Table 77.5 present a list of freely available software for the solution of linear algebra problems. 
The interest is in software for high-performance computers that is available in “open source” form on 
the Web for solving problems in numerical linear algebra, specifically dense, sparse direct and iterative 
systems, and sparse iterative eigenvalue problems. 

Additional pointers to software can be found at: 
www.nhse.org/rib/repositories/nhse/catalog/#Numerical_Programs_and_ 
Routines. 

A survey of Iterative Linear System Solver Packages can be found at: 
www.netlib.org/utk/papers/iterative-survey. 


TABLE 77.1 Support Routines for Numerical Linear Algebra 


Type Language Mode 
Package Support Real Complex 77 c c++ Seq _ Dist Dense Sparse 
ATLAS yes x x xX XxX x x 
BLAS yes x x xX XxX x x 
FLAME yes x x xX XxX x x 
LINALG 2 
MTL yes Xx Xx X X 
NETMAT yes x x x x 
NIST S-BLAS yes x x xX XxX x x 
PSBLAS yes x x xX XxX x M x 
SparseLib++ yes x x >.< >.< X xX 
uBLAS yes x Xx x x X X 


77-1 


77-2 Handbook of Linear Algebra 


TABLE77.2 Available Software for Dense Matrix 


Type Language Mode 
Package Support Real Complex f77 c  ct+ Seq Dist 
LAPACK yes x x xX XxX xX 
LAPACK95 yes x x 95 xX 
NAPACK yes x x 
PLAPACK yes x x X XxX M 
PRISM yes x x xX M 
ScaLAPACK yes x x x XxX M/P 
TABLE 77.3 Sparse Direct Solvers 
Type Language Mode 
Package Support Real Complex f77  c c++ Seq Dist SPD Gen 
DSCPACK yes x x xX M x 
HSL yes x x xX xX x x 
MFACT yes x x xX x 
MUMPS yes x xX xX XxX xX M x xX 
PSPASES yes x xX XxX M x 
SPARSE 2 x x x xX x x 
SPOOLES 2 x x x xX M x 
SuperLU yes Xx Xx xX XxX X M Xx 
TAUCS yes x xX x xX x xX 
UMFPACK yes x xX x xX x 
Y12M 2 x xX xX x x 
TABLE 77.4 Sparse Eigenvalue Solvers 
Type Language Mode 
Package Support Real Complex f77 c c++ Seq Dist Sym Gen 
(B/H)LZPACK yes x xX x X  M/P xX x 
HYPRE yes x x xX M xX 
QMRPACK 2 x xX x xX xX x 
LASO 2 x x xX xX 
P_ARPACK yes x xX x xX  M/P x 
PLANSO yes x x xX M xX 
SLEPc yes x xX x xX M xX x 
SPAM yes Xx 90 Xx Xx 
TRLAN yes x x xX M xX 
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Language Mode Precond. Iterative Solvers 

Package Support — Real Comp. £77 c ct+ Seq Dist SPD Gen SPD Gen 
AZTEC no x x x M x x x x 
BILUM yes x x x M x x 
BlockSolve95 ? ».< xX xX x M x x x x 
BPKIT yes x xX xX x x M x x 

CERFACS yes x x x xX M x xX 
HYPRE yes x xX xX x M x x x x 
IML++ 2 x xX xX x x x x 
ITL yes x x x x x 
ITPACK ? x x x x x 
LASPack yes x Xx Xx Xx xX 
LSQR yes x xX xX x x 
pARMS yes x x XxX Xx M Xx X 
PARPRE yes x x M x ».« 

PETSc yes x x xX xX x M x x x x 
P-SparsLIB yes x x M Xx X 
PSBLAS yes x x f90 XxX M x x x x 
QMRPACK ? x x x x x 
SLAP ? x ».< x 

SPAI yes x x x M x x 
SPLIB 2 x x x x x 
SPOOLES 2 x x x x x x x xX 
SYMMLQ yes x x x x x x 
TAUCS yes x x x x x x x x 
Templates yes ».« xX xX >.< Xx X 
Trilinos yes ».« xX Xx M xX >.< ».« Xx 


Glossary 


This glossary covers most of the terms defined in Chapters 1 to 49. It does not cover some terminology 
used in a single chapter (including most of the terminology that is specific to a particular application 
(Chapters 50 to 70)), nor does it cover most of the terminology used in computer software (Chapters 71 
to 77). When two sections are listed, both define the term; the first listed chapter is the first instance, the 
second is the primary chapter dealing with the topic. Definitions in this glossary may not give all details; 
the reader is advised to consult the chapter/section following the term and detinition). 


A 


abelian (group G): A commutative group, i.e., ab = ba for all a,b € G. Preliminaries 

absolute (matrix norm): As a vector norm, each member of the family is absolute. 37.3 

absolute (vector norm || - ||): For all vectors x, || |x| || = ||x||. 37.1 

absolute algebraic connectivity (of simple graph G = (V, E)): maxa(G,,) where the maximum is taken 
over all nonnegative weights w of the edges of G such that )0,., w(e) = |E|. 36.5 

absolute error (in approximation Z to z): ||z — || 37.4 

absolute value (of complex number a + bi): /a* + b?. Preliminaries 

absolute value (of real matrix A): The nonnegative matrix obtained by taking element-wise absolute values 
of A’s entries. 9.1 

access: Vertex u has access to vertex v in a digraph I if there is a walk in’ from u to 1; also applied to sets 
of vertices. 9.1, 29.5 

access equivalence class (of a digraph): An equivalence class under the equivalence relation of access. 
9.1, 29.5 

access equivalence class (of a matrix): An access equivalence class of its digraph. 9.1 

access equivalent: Two vertices in a digraph that have access to each other. 9.1, 29.5 

active branch (at a Type I characteristic vertex of a tree): For some Fiedler vector y the entries in y 
corresponding to the vertices in the branch are nonzero. 36.3 

acyclic (square matrix A): The graph of A has no cycles. 19.3 

additive (map @ : F"*" > F'*"): (A+ B) = $(A) + @(B), forall A, B € F™*". 22.4 

additive coset (of subspace W): A subset of vectors the form v + W = {v+w:we W}.2.3 

additive D-stable (real square matrix A): A + D is stable for every nonnegative diagonal matrix D. 19.4 
additive inverse eigenvalue problem (AIEP): Given A € C”*" andAj,... ,A, € C find a diagonal matrix 
De C"*" such that 0(A + D) = {Aj,... An}. 20.9 

additive preserver: An additive map preserving a certain property. 22.4 

adjacency matrix (of a digraph I’ of order n): The n x n 0, 1-matrix whose j, jth entry is 1 if there is an 
arc from the ith vertex to the jth vertex and 0 otherwise. 29.2 

adjacency matrix (of a graph G of order n): The symmetric n x n matrix whose i, jth entry is equal to 
the number of edges between the ith vertex and the jth vertex. 28.3 

adjacent (matrices A, B): rank (A — B) = 1.22.4 

adjacent (vertices u and v in a graph): There exists an edge with endpoints u and v. 28.1 

adjoint (of a matrix): See Hermitian adjoint. 

adjoint (of a linear operator T on an inner product space V): (T(u), v) = (u, T*(v)) for allu,v € V.5.3 
adjugate: The transpose of the matrix of cofactors. 4.2 

affine parameterized inverse eigenvalue problem: See Section 20.9. 

AIEP: See additive inverse eigenvalue problem. 
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algebra (associative): A vector space A over a field F with a bilinear multiplication (x,y) > xy from 
Ax Ato A satisfying (xy)z = x(yz). Preliminaries, 69.1 

algebra (nonassociative): See Section 69.1. 

algebraic connectivity (of simple a graph): The second least Laplacian eigenvalue. 28.4, 36.1 

algebraic multigrid preconditioner: A preconditioner that uses principles similar to those used for PDE 
problems on grids, when the “grid” for the problem is unknown or nonexistent and only the matrix is 
available. 41.4 

algebraic multiplicity (of an eigenvalue): The number of times the eigenvalue occurs as a root in the 
characteristic polynomial of the matrix. 4.3 

algorithm: A precise set of instructions to perform a task. 37.7 

allows: If P is a property referring to a real matrix, then a sign pattern A allows P if some real matrix in 
Q(A) has property P. 33.1 

allows a properly signed nest (n x n sign pattern A): There exists B € Q(A) and a permutation matrix 
P such that sgn(det(P? B P[{1, ...k}])) = (—1)* for k= 1,...,n. 33.4 

Alt: the map Alt(v) @ --- @ v;) = + Denes, Sgn(I)Vz(1) @ +++ @ Vx (ky). 13.6 

alt multiplication: (v; A --- A Vp) A (Vp41 A+++ A Vp4q) = V1 A+++ A Vp4q- 13.7 

alternate path to a single arc (in a digraph): A path of length greater than 1 between vertices i and j such 
that the arc (7, 7) is in the digraph. 35.7 

alternating (bilinear form f): f(v,v) = 0 for allv € V. 12.3 

alternating (n x n matrix A): aj; = 0, i = 1,2,...,nandajj = —ajj, 1 <i,j <n.123 

alternating (multilinear form): See antisymmetric. 

alternating algebra: See Grassman algebra. 

alternating product: See exterior product. 

alternating space: See Grassman space. 

alternator: See Alt. 

angle (between two nonzero vectors u and vin a real inner product space): The real number 6,0 < 6 < z, 
such that (u, v) = ||ul|||v|| cos. 5.1 

annihilator: The set of linear functionals that annihilate every vector in the given set. 3.8 


antidiagonal (of n x n matrix A): The elements aj,_;,i = 1,...,k — lwith2 <k <n+1. 48.1 
anti-identity matrix: The 1 x nm matrix with ones along the main antidiagonal, ie., diy4i-; = 1, 
i = 1,...,n and zeros elsewhere. 48.1 

antisymmetric (bilinear form f): f(u,v) = — f(v,u) for all u,v € V. 12.3 

antisymmetric (multilinear form): A multilinear form that is an antisymmetric map. 13.4 
antisymmetric (map w € L”(V;U)): W(Wa(1)>--- > Va(m) = sgn(w)W(v1,...5Vm) for all permuta- 


tions 7. 13.4 

aperiodic (matrix): An irreducible nonnegative matrix of period 1. 9.2 

appending G, at vertex v of G;: Constructing a simple graph from G, U G, by adding an edge between 
v and a vertex of G». 36.2 

Arbitrary Precision Approximating (APA) algorithms: See Section 47.4. 

arc: An ordered pair of vertices (part of a directed graph). 29.1 

argument (of a complex number): 6 in the form re’? with 0 < 6 < 2z. Preliminaries 

Arnoldi factorization (k-step): AV; = Vi Hy +fre; where Vi € C"** has orthonormal columns, Vif, = 0 
and Hy = V,;*AV; isak x k upper Hessenberg matrix with nonnegative subdiagonal elements. 41.3, 44.2 
associated undirected graph (of a digraph T = (V, E)): The undirected graph with vertex set V, having 
an edge between vertices u and v if and only if at least one of the arcs (u,v) and (v, u) isin’. 29.1 
associates (in a domain): a, b are associates if if a|b and ba. 23.1 

association scheme: A set of graphs Go,... , Gq on a common vertex set satisfying certain axioms. 28.6 
associative algebra: See algebra (associative). 

asymmetric (digraph T = (V, E)): (i, 7) € E implies (j,i) ¢ E for all distinct i,j € V. 35.1 
augmented matrix (of a system of linear equations): Matrix obtained by adjoining the constant column 
to the coefficient matrix. 1.4 
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B 


backward error (for f(x): A vector e € R" of smallest norm for which f(x +e) = F(x). 37.8 
backward stable (algorithm): The backward relative error e exists and is small for all valid input data x 
despite rounding and truncation errors in the algorithm. 37.8 

badly conditioned: See ill-conditioned. 

balanced ((0, 1)-design matrix W): Every row of m x n matrix W has exactly (n + 1)/2 ones if is n odd; 
exactly 1/2 ones or exactly (n + 2)/2 ones if n even. 32.4 

balanced (real or sign pattern vector): It is a zero vector or it has both a positive entry and a negative 
entry. 33.3 

balanced column signing (of matrix A): A column signing of A in which all the rows are balanced. 33.3 
balanced row signing (of matrix A): A row signing of A in which all the columns are balanced. 33.3 
banded (family of Toeplitz matrices with symbol a): There exists some m > 0 such that ai, = 0 for all 
k > m. 16.2 

barely L-matrix: An L -matrix that is not an L-matrix if any column is deleted. 33.3 

bases: Plural of basis. 

basic class (of square nonnegative matrix P ): An access equivalence class B of P with p(P[B]) = p. 9.3 
basic reduced digraph (of square nonnegative matrix P ): The digraph whose vertex-set is the set of basic 
classes of P and whose arcs are the pairs (B, B’) of distinct basic classes of P for which there exists a simple 
walk from B to B’ in the reduced digraph of P. 9.3 

basic solution (to a least squares problem): A least squares solution with at least n — rankA zero compo- 
nents. 39.1 

basic variable: A variable in a system of linear equations whose value is determined by the values of the 
free variables. 1.4 

basis: A set of vectors that is linearly independent and spans the vector space. 2.2 

BD: See Bezout domain. 

Bezout domain (BD): A GCCD D such that for any two elements a,b € D, (a,b) = pa + qb, for some 
pq €D.23.1 

biacyclic (matrix): A matrix whose sparsity pattern has an acyclic bipartite graph. 46.3 

biadjacency matrix (of bipartite graph G): A matrix with rows indexed by one of the bipartition sets and 
columns indexed by the other, with the value of the entry being the number of edges between the vertices 
(or weight of the edge between the vertices). 30.1 

BiCG algorithm: Iterative method for solving a linear system using non-Hermitian Lanczos algorithm; 


the approximate solution x; is chosen so that the residual r; is orthogonal to 

Span{fo, A*fo,... , A**l¥o}. 41.3 

BiCGSTAB algorithm: Iterative method for solving a linear system using non-Hermitian Lanczos algo- 
rithm with improved stability. See Section 41.3. 

biclique (of a graph): A subgraph that is a complete bipartite graph. 30.3 

biclique cover (of graph G = (V, E)): A collection of bicliques of G such that each edge of E is in at least 
one biclique. 30.3 

biclique cover number (of graph G): The smallest k such that there is a cover of G by k bicliques. 30.3 
biclique partition (of G = (V, E)): A collection of bicliques of G whose edges partition E. 30.3 
biclique partition number (of graph G): The smallest k such that there is a partition of G into k bi- 
cliques. 30.3 

bidual space: The dual of the dual space. 3.8 

big-oh: function f is O(g) if beyond a certain point | f| is bounded by a multiple of |g|. Preliminaries 
bigraph (of the m x n matrix A): The simple graph with vertex set U U V where U = {1,2,... , m} and 
V = {I',2’,...,n'}, and edge set {{i, j’} : aij A O}. 30.2 

bilinear form (on vector space V over field F ): Amap f from V x V into F that satisfies f (au,-+bu, v) = 
af (uj,v) + bf (uz, v) and f(u,av, + bv2) = af (u,v1) + bf (u, v2). 12.1 

bilinear noncommutative algorithms: See Section 47.2. 

binary matrix: A (0,1)-matrix, i.e., a matrix in which each entry is either 0 or 1. 31.3 
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biorthogonal (sets of vectors {v,,...,vx} and {wi,..., Wx} in an inner product space): (vj, wj) = 0 if 
ix j.41.3 

bipartite (graph G): The vertices of G can be partitioned into disjoint subsets U and V such that each 
edge of G has the form {u,v} where u € U and v € V. 28.1, 30.1 

bipartite fill-graph of n x n matrix A with each diagonal entry nonzero: The simple bipartite graph with 
vertex set {1,2,...,n}U{1’,2’,... ,n’} with an edge joining i and j’ if and only if there exists a path from 
i to j in the directed graph, I'(A), of A each of whose intermediate vertices has label less than min{i, 7}. 30.2 
bipartite graph (of sparsity pattern S): The graph with vertices partitioned into row vertices r1,... Tm 
and column vertices c),... ,C,, where rz and r; are connected if and only if (k,1) € S. 46.3 

bipartite sign pattern: A sign pattern whose digraph is bipartite. 33.5 

bipartition: The sets into which the vertices of a bipartite graph are partitioned. 30.1 

block (in a 2-design): A subset B; of X (see 2-design). 32.2 

block (of a graph or digraph): A maximal nonseparable sub(di)graph. 35.1 

block (of a matrix): An entry of a block matrix, which is a submatrix of the original matrix. 10.1 
block-clique (graph or digraph): Every block is a clique. 35.1 

block diagonal (for a particular block matrix structure): A square block matrix A with all off-diagonal 
blocks 0. 10.2 

block matrix: A matrix that is partitioned into submatrices with the row indices and column indices 
partitioned into consecutive subsets sequentially. 10.1 

block lower triangular (matrix A): A’ is block upper triangular. 10.2 

block—Toeplitz matrix: A block matrix A = [Aj;] such that Aj4,; = A”), 48.1 
block—Toeplitz—Toeplitz—block (BTTB) matrices: a block—Toeplitz matrix in which the blocks A“ are 
themselves Toeplitz matrices. 48.1 

block upper triangular (for a particular block matrix structure): A block matrix having every subdiagonal 
block 0. 10.2 

(0,1)-Boolean algebra: {0, 1}, withO+0=0,0+1=1=1+40,14+1=1,0*1=0=1%0,0*«0=0, 
and 1*«1= 1.30.3 

Boolean matrix: A matrix whose entries belong to the Boolean algebra. 30.3 

Boolean rank (of m x n Boolean matrix A): The minimum k such that there exists an m x k Boolean 
matrix B andak x n Boolean matrix C such that A = BC. 30.3 

Bose—Mesner algebra: The algebra generated by the adjacency matrices of the graphs of an association 
scheme. 28.6 

bottleneck matrix (of a branch of tree T at vertex v): The inverse of the principal submatrix of the 
Laplacian matrix corresponding to the vertices of that branch. 36.3 

boundary (ofa subset S of R or C): The intersection of the closure of S and the closure of the complement 
of S. Preliminaries 

branch (of tree T at vertex v): A component of T — v. 34.1 

Bunch-Parlett factorization (of symmetric real matrix H): The factorization P’ HP = LBL", where 
P isa permutation matrix, B is a block—diagonal matrix with diagonal blocks of sizes 1 x 1 or 2 x 2, and 
L isa full column rank unit lower triangular matrix, where the diagonal blocks in L which correspond to 
2 x 2 blocks in B are 2 x 2 identity matrices. 15.5 

Businger—-Golub pivoting: a particular pivoting strategy for QR-factorization. 46.2 


C 


canonical angles between the column spaces of X, Y ¢ C"**:6; = arccos o;, where {a;}*_, are the singular 
values of (Y*Y)~'/2Y¥* X(X*X)~!/? 15.1 (equivalent to principal angles 17.1) 


canonical angle matrix: diag(0),62,... , 0%), where 0), 62,... , 0% are the canonical angles. 15.1 
canonical correlations (between subspaces X and Y of C’): Cosines of principal (canonical) angles 17.7 
Cauchy matrix: Given vectors x = [x),...,X,]7 andy = [y1,...; yn], the Cauchy matrix C(x, y) has 


i, j-entry equal to rao 48.1 
ivy; 


Glossary G-5 


center See central vertex. 

central (real matrix B): The zero vector is in the convex hull of the columns of B. 33.11 

central vertex (of a generalized star): a vertex such that each of its neighbours are pendant vertices of their 
branches, and each branch is a path. 34.1 

CG: See conjugate gradient algorithm. 

CGS algorithm: Iterative method for solving a linear system using non-Hermitian Lanczos algorithm. 
See Section 41.3. 

change-of-basis matrix (from B to C): The matrix consisting of the coordinate vectors with respect to 
basis C of the basis vectors in 3. 2.6 

characteristic equation (of the pencil A — xB): det(xB — A) = 0. 43.1 

characteristic polynomial (of matrix A): det(xI — A). 4.3 

characteristic polynomial (of the pencil A — xB): det(xB — A). 43.1 

characteristic polynomial (of a graph): The characteristic polynomial of its adjacency matrix. 28.3 
characteristic vector (of a subset of m vertices of a graph): The m-vector whose ith entry is 1 ifi € X, 
and 0 otherwise. 30.3 

characteristic vertex (of tree T): See Section 36.3. 

Cholesky decomposition (of positive-definite matrix A): A = G G* with G € C”*” lower triangular and 
having positive diagonal entries. 38.5 

Cholesky factorization: See Cholesky decomposition. 

chord (of a cycle in a graph): An edge joining two nonconsecutive vertices on the cycle. 30.1 

chordal bipartite graph: Every cycle of length 6 or more has a chord. 30.1 

chordal graph: Every cycle of length 4 or more has a chord. 30.1 

1-chordal: See block-clique. 

chromatic index (of graph G): The smallest number of classes in a partition of edges of G so that no two 
edges in the same class meet. 27.6 

chromatic number (of graph G): The smallest number of color classes of any vertex coloring of G (not 
defined if G has loops). 28.5 

circulant matrix: A Toeplitz matrix in which every row is obtained by a single cyclic shift of the previous 
row. 20.3, 48.1 

clique (of a digraph): Every vertex has a loop and for any two distinct vertices u, v, both arcs (u,v), (v, u) 
are present. 35.1 

clique (ofa graph that allows loops but not multiple edges): Every vertex has a loop and for any two distinct 
vertices u,v, the edge {u, v} is present. 35.1 

clique (of a simple graph): A subgraph isomorphic to a complete graph. 28.5 

clique number (of graph G): The largest order of a clique in G. 28.5 

closed cone: A cone that is a closed subset of the vector space. 8.5, 26.1 

closed under matrix direct sums (class of matrices X): Whenever Ay, Az,..., Ag are X-matrices, then 
A; ® Ar ®--:@ Ax is an X-matrix. 35.1 

closed under permutation similarity (class of matrices X): Whenever A is an X-matrix and P is a 
permutation matrix, then P T AP isan X-matrix. 35.1 

closed under taking principal submatrices: See hereditary. 

closed walk (in a digraph): A walk in which the first vertex equals the last vertex. 54.2 

4-cockade: A bipartite graph created by adding 4 cycles through edge identification to a 4-cycle. 30.2 
cocktail party graph: The graph obtained by deleting a disjoint edges from the complete graph K2,. 28.2 
coclique (of a graph): An induced subgraph with no edges. 28.5 

co-cover (ofa (0, 1)-matrix): A collection of 1s such that each line of A contains at least one of the 1s. 27.1 
co-index (of square nonnegative matrix P): max{vp(A):4 € o(P),|A| = panda ¥ p}. 9.3 

codomain (of T: V > W): The vector space W. 3.1 

coefficient matrix: The matrix of coefficients of a system of linear equations. 1.4 

coefficients (of a linear equation): The scalars that occur multiplied by variables. 1.4 

i, j-cofactor: (—1)'+/ times the i, j-minor. 4.1 
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Colin de Verdiére parameter: A parameter of a simple graph. 28.5 

color class: A coclique in a vertex coloring partition of the vertices of a graph. 28.5 

column: The entries of a matrix lying in a vertical line in the matrix. 1.2 

column equivalent (matrices A, B €¢ D”*"): B = AP for some D-invertible P. 23.2 

column signing (of (real or sign pattern) matrix A): AD where D isa signing. 33.3 

column space: See range. 

column-stochastic (matrix): A square nonnegative matrix having all column sums equal to 1. 9.4 
column sum vector (of a matrix): The vector of column sums. 27.4 

combinatorially orthogonal (sign pattern vectors x,y): |{i : xiv; 4 0}| # 1.33.10 

combinatorially symmetric (partial matrix B): b;; specified implies bj; specified. 35.1 

combinatorially symmetric sign pattern A: a;; 4 0 if and only if aj; 4 0. 33.5 

communicate: See access equivalent. 

commute: Matrices or linear operators A, B such that AB = BA. 1.2, 3.2 

companion matrix (of a monic polynomial): Square matrix with ones on the subdiagonal and last column 
consisting of negatives of the polynomial coefficients 4.3, 6.4 

comparison matrix (of real square matrix A): Matrix having i, j-entry —|a;;| fori A j and i,i-entry 
|Aij |. 19.5 

compatible (generalized sign patterns): The intersection of their qualitative classes is nonempty. 33.1 
complement (of set X in universe S): The elements of S not in X. Preliminaries 

complement (of binary matrix M): J — M (where J is the all 1s matrix). 31.3 

complement (of simple graph G = (V,E)): The simple graph having vertex set V and as edges all 
unordered pairs from V that are not in E. 28.1 

complement (orthogonal): See orthogonal complement. 

complete (bipartite graph): A simple bipartite graph with bipartition {U, V} such that each {u,v} is an 
edge (for all u € U, v € V). 28.1, 30.1 

complete (simple graph): The edge set consists of all unordered pairs of distinct vertices. 28.1 

complete orthonormal set: An orthonormal set of vectors whose orthogonal complement is 0. 5.2 
completed max-plus semiring: The set R U {00} equipped with the addition (a,b) +» max(a, b) and 
the multiplication (a,b) a + b, with the convention that —oo + (+00) = +00 + (—00) = —00. 25.1 
completely positive (matrix A): A = CC! for some nonnegative n x m matrix C. 35.4 

completely reducible (matrix A): There is a permutation matrix P suchthat PAP? = A;@A)®---@ Ax 
where Aj, Ao,... , Ag are irreducible and k > 2. 27.3 

completion (of a partial matrix): A choice of values for the unspecified entries. 35.1 

X-completion property (where X is a type of matrix): Digraph (respectively, graph) G has this property 
if every partial X-matrix B such that D(B) = G (respectively, 9(B) = G) can be completed to an 
X-matrix. 35.1 

complex conjugate (of a + bi € C): a — bi. Preliminaries 

complex sign pattern matrix: A matrix of the form A = A, + 7A) for sign patterns A, and A). 33.8 
complex vector space: A vector space over the field of complex numbers. 1.1 

component-wise relative backward error of the linear system: See Section 38.1. 


component-wise relative distance (between matrices H and H): reldist(H, H) = max ea 
ij ij 

0/0 = 0. 15.4 

composite cycle (in sign pattern A): A product of disjoint simple cycles. 33.1 

k x k-compound matrix: A matrix formed from the k x k minors of a given matrix. 4.2 


condensation digraph: See reduced digraph. 
condition number (of z with respect to problem P): 


condp(z) = lim,_,9 sup (east Pe) (#4) \||Sz|| < © > .37.4 


condition number (of an eigenvalue): See individual condition number. 
condition number (of matrix A for linear systems): (A) = ||A7"||y || Ally. 37.5, 38.1 
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condition number of the linear system: AX = b: « (A,X) = ||A7}|| tt. 38.1 

condition number (of least squares problem Ax = b): «(A) = o;/op, where rank A = p. 39.6 
condition numbers for polar factors (A = U P) in the Frobenius norm: 

conds(X) = lims_,9 SUP} A Ally <3 1erMe for X = P or U. 15.3 

conductance: A parameter of a simple graph. 28.5 

cone: A subset K of a real or complex vector space such that for each x,y € K,c > 0, x+y € K and 
cx € K;in Section 26.1 cone is used to mean proper cone. 8.5, 26.1 

conformable: See conformal. 

conformal (partitions of matrices A, B): Partitions of block matrices that allow multiplication via the 
block structure. 10.1 

congruent (square matrices A,B over field F): There is an invertible matrix C such that B = C os 
AC. 12.1 

*congruent (square matrices A, B over C): There is an invertible matrix C such that B = C* AC. 8.3 
g-congruent (square matrices A, B over field F with automorphism g): There is an invertible matrix C 
such that B = C! Ag(C),. 12.4 

conjugate: See complex conjugate. 

conjugate gradient (CG) algorithm (to solve Hx = b for preconditioned Hermitian positive definite 


matrix H): At each step k, the approximation x; of the form x, € Xo + Span{ro, Hro,..., H*-!40} for 
which the ¥ H-norm of the error, ||ex|| yq@ = (ex; He,)'/2, is minimal. 41.2 
conjugate partition (of s sequence of positive integers (1, U2,...,U,)): The ith element of the conjugate 


partition is the number of js such that u; > i. Preliminaries 

connected (graph): A graph with nonempty vertex set such that there exists a walk between any two distinct 
vertices. 28.1 

connected (digraph): A digraph whose associated undirected graph is connected. 29.1 

connected component (ofa graph): A connected (induced) subgraph not properly contained ina connected 
subgraph. 28.1 

consecutive ones property (of a (0, 1)-matrix A): There exists a permutation matrix P such that P A isa 
Petrie matrix. 30.2 

consistent: A system of linear equations that has one or more solutions. 1.4 

k-consistent sign pattern A: Every matrix B € Q(A) has exactly k real eigenvalues. 33.5 

constant (of a linear equation): The scalar not multiplied by a variable. 1.4 

constant vector: The vector of constants of a system of linear equations. 1.4 

contraction (matrix): A matrix A € C”*” such that || Al|z < 1. 18.6 

contraction (of edge e in graph G = (V, E)): The operation that merges the endpoints of e in V, and 
deletes e from E. 28.2 

convergent (square nonnegative matrix P): limy,.5. P” = 0.9.3 

convergent regular splitting: Square real matrix A has a convergent regular splitting if A has a represen- 
tation A = M — N such that N > 0, M invertible with M~! > 0 and M™!N is convergent. 9.5 
converges geometrically to a with (geometric) rate 8 (complex sequence {4} m=0,1,...): a m= 
0,1,...} is bounded for each B < y < 1.9.1 

convex (set of vectors): Closed under convex combinations. Preliminaries 

convex combination (of vectors vj, V2,... , vx): A vector of the form a,v,; + ayv2 +--+ + agvy with a; 
nonnegative and }> a; = 1 (vector space is real or complex). Preliminaries 

convex cone: See cone. 

convex function: A function f : S — R where S is a convex set and for all a € R such that0 <a < 1 
andx,yé S, f(ax+(1—a)y) < af(x) + (1 —a) f(y). Preliminaries 

convex hull (of a set of vectors): The set of all convex combinations of the vectors. Preliminaries 

convex polytope: The convex hull of a finite set of vectors in R”. Preliminaries 

coordinate (part of a vector): In the vector space F”, one of the entries of a vector. 1.1 

coordinate mapping: The function that maps a vector to its coordinate vector. 2.6 

coordinate vector: The vector of coordinates of a vector relative to an ordered basis. 2.6 
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coordinates (of a vector relative to a basis): The scalars that occur when the vector is expressed as a linear 
combination of the basis vectors. 2.6 

copositive (matrix A): x’ Ax > 0 for all x > 0. 35.5 

coprime (elements in a domain): 1 is a greatest common divisor of the elements. 23.1 

corner minor: The determinant of a submatrix in the upper right or lower left corner of a matrix. 21.3 
correlation matrix: A positive semidefinite matrix in which every main diagonal entry is 1. 8.4 

cosine (of a complex square matrix A): The matrix defined by the cosine power series, 

cos(A) = 1 — Ape. + GOA yo 15 

cospectral (graphs): Having the same spectrum. 28.3 

cover (of a (0, 1)-matrix): A collection of lines that contain all the 1s of the matrix. 27.1 

curve segment (from x to y): The range of a continuous map @ from [0,1] to R” with @(0) = x and 
o(1) = y. 28.2 

cut-vertex (of connected simple graph G): A vertex v of G such that G — v is disconnected. 36.2 
(k-)cycle (in a graph or digraph): A walk (of length k) with all vertices distinct except the first vertex equals 
the last vertex. 28.1, 29.1 

cycle (permutation): A permutation t that maps a subset to itself cyclically. Preliminaries 

k-cycle (in sign pattern A): A formal product of the form aj,;,4;,;, . .. di,i,, where each of the elements is 
nonzero and the index set {i),i2,... , iz} consists of distinct indices. 33.1 

n-cycle matrix: The n x n matrix with ones along the first subdiagonal and in the 1,n-entry, and zeros 
elsewhere. 48.1 

n-cycle pattern: A sign pattern A where the digraph of A is an n-cycle. 33.5 

cycle-clique (digraph): The induced subdigraph of every cycle is a clique. 35.7 

cycle product: A walk product where the walk is a cycle. 29.3 

cyclic normal form (of matrix A): A matrix in specific form that is permutation similar to A. 29.7 
cyclically real (square ray pattern A): The product of every cycle in A is real. 33.8 


D 


A(G): max(p — q) over all ways in which q vertices may be deleted from graph G leaving p 
paths. 34.2 

D-invertible, D-module, D-submodule: Alphabetized under invertible, module, submodule. 

D-optimal (design matrix W): det W7 W is maximal over all (+1)- (or (0, 1)-)matrices of the given size. 
32.1 

damped least squares solution: The solution to the problem (A7A + aI)x = A’b, where @ > 0. 39.8 
data fitting: See Section 39.2. 

data perturbations (for linear system Ax = b): Perturbations of A and/or b. 38.1 

decomposable tensor: A tensor of the form v; ® --- ® Vin. 13.2 

defective (matrix A ¢ F"*”): There is an eigenvalue of A (over algebraic closure of F ) having geometric 
multiplicity less than its algebraic multiplicity. 4.3 

deflation: A process of reducing the size of the matrix whose eigenvalue decomposition is to be determined, 
given that one eigenvector is known. 42.1 

degenerate (bilinear form f on vector space V): Not nondegenerate, i.e., the rank of f less than dim V, 12.1; 
also applied to g-sesquilinear form 12.4. 


degree (of polynomial p(x;,...,x,) 4 0): The largest integer m such that there is a term cx"... x7 
with c # 0 in p having degree m. 23.1 
degree (of term cxy" ...x/"" in polynomial p(x),...,x;,)): The sum of the degrees of the x;, Le., Zjmj. 


degree (of a vertex v): The number of times that v occurs as an endpoint of an edge. 28.1 

deletion (of edge e from graph G = (V, E)): The operation that deletes e from E. 28.2 

deletion (of vertex v from graph G = (V, E)): The operation that deletes v from V and all edges with 
endpoint v from E. 28.2 

depth (of eigenvector x for eigenvalue 4 of A): The natural number h such that 

x € range(A — AI)" — range(A — AI)"*+!. 6.2 
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derogatory (matrix A €¢ F"*"): Some eigenvalue of A (over algebraic closure of F ) has geometric mul- 
tiplicity greater than 1. 4.3 

2-design with parameters (v,k,A): A collection of k-element subsets (blocks) B; of a finite set X with 
|X| = v, such that each 2-element subset of X is contained in exactly A blocks. 32.1 

(+1)-design matrix: A matrix whose entries are 1 or —1. 32.1 

(0, 1)-design matrix: A matrix whose entries are 1 or 0. 32.1 

det: See determinant. 

determinant (det): A scalar-valued function of a square matrix or linear operator, defined inductively, 
and also given by det(A) = des: sgn o [ [jy Gigi) 4:1 

determinantal region (of complex sign pattern or ray patern A): S4 = {det(B) : B € Q(A)}. 33.8 
determined by its spectrum (graph G): Every graph cospectral with G is isomorphic to G. 28.3 
diagonal (of a matrix): The diagonal means the main diagonal, i.e., the set of diagonal entries of a matrix, 
1.2. A diagonal is a collection of n nonzero entries in A no two on the same line. 27.2 

diagonal entry: An entry that lies in row k and column k for some k. 1.2 

diagonal matrix: A matrix all of whose off-diagonal entries are zero. 1.2, 10.2 

diagonal pattern: A square sign pattern all of whose off-diagonal entries are zero. 33.1 

diagonal product (of a matrix): The product of the entries on a diagonal of the matrix. 27.6 
diagonalizable (matrix A € F"*"): There exist nonsingular matrix B € F"*" and diagonal matrix 
D ¢ F"*" such that A= BDB™!. 4.3 

diagonally dominant (n x n complex matrix A): |a;i| > ia jxi |a;;| fori = 1,...,n. 9.5 

diagonally scaled representation (of symmetric matrix H with positive diagonal entries): A factored 
representation H = DAD with D = diag(/hi1,... 5 Rnn)s 4ij = we 15.4 

diagonally scaled totally unimodular (DSTU) (real matrix A): There exist aidgonal matrices D,, Dy and 
totally unimodular Z such that A = D, ZD). 46.3 

diagonally stable: See Lyapunov diagonally stable. 

diameter (of a connected graph): The largest distance that occurs between two vertices. 28.1 

digraph: A directed graph having at most one arc between each ordered pair of vertices (loops permit- 
ted). 29.1 

digraph (of n x n matrix A): The digraph having vertices {1,...,n} and having arc (i, j) exactly when 
ai; # 0.9.1, 29.2 

digraph (of n x n partial matrix B): The digraph having vertices V = {1,...,n} and for each i,j € V, 
(i, j) is an arc exactly when bj; is specified. 35.1 

dilation: Matrix A € C”*" has a dilation B € C”™*™ if there is X € C”™*” such that X*X = I, and 
X*BX = A. 18.6 

dim: See dimension. 

dimension (dim) (of a vector space): The number of vectors in a basis for the vector space. 2.2 
dimension (dim) (of a convex polytope): The smallest dimension of an affine space containing it. 27.6 
direct sum (of matrices): A block diagonal matrix with these matrices as the diagonal blocks. 2.4, 10.2 
direct sum (of subspaces): W; + --- + W, is direct iffor alli = 1,...,k, WiN 3s Wj = {0}. 2.3 

direct sum (of vector spaces): See external direct sum. - 

directed bigraph (of the real m x n matrix A): The directed graph with vertices 1,2,... ,m, 1’,2',... ,n’, 
with arc (i, j’) if and only if a;; > 0, and with arc (j’,7) if and only if a;; < 0. 30.2 

directed edge: See arc. 

directed graph: A finite nonempty set of vertices and a finite multiset of arcs, where each arc is an ordered 
pair of vertices. 29.1 

directed multigraph: A directed graph having more than one arc between at least one pair of 
vertices. 29.1 

disjoint (graphs): The vertex sets are disjoint sets. 28.1 

dispersion: See Section 21.3. 

distance (between two vectors): The norm of their difference. 5.1 
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distance (between two vertices in a graph): The length of a shortest path between the vertices (infinite if 
no path). 28.1 

distance of W(A) to the origin: w(A) = min{|| : 4 € W(A)}. 18.1 

distance-regular graph: A connected graph with the property that for every pair of vertices u, v, the num- 
ber of vertices that have distance i from u and distance j from v depends only oni, j, and the distance 
between u and v. 28.6 

distinguished eigenvalue (of nonnegative square matrix P): An eigenvalue of P that has a semipositive 
eigenvector. 9.1 

d divides a (ina domain D): a = db for some b € D. 23.1 

divisor (i.e., d is a divisor of a): See divides. 

domain (ring): An integral domain, i.e., a commutative ring without zero divisors and containing identity 
1. 23.1 

domain (of T: V + W): The vector space V. 3.1 

dominant eigenvalue: A, where |A;| > |A2| > --- > |A,| are the eigenvalues. 42.1 

dot product: See standard inner product. 

double echelon form: A special type of real matrix, see 21.1 

double generalized star: A tree constructed by joining the centers of two generalized stars by an edge. 34.7 
double path: A tree constructed by joining two degree two vertices of two paths by an edge. 34.7 

double precision: Typically, floating-point numbers have machine epsilon roughly 10~'° and precision 
roughly 16 decimal digits. 37.6 

doubly directed tree: A digraph I whose associated undirected graph is a tree and whenever (i, ) is an 
arc in’, (j,7) is also an arc in. 29.1 

doubly nonnegative (matrix): A positive semidefinite matrix having every entry nonnegative. 35.4 
doubly stochastic (matrix): A square nonnegative matrix having all row and column sums equal 
to 1.9.4 

downdating (QR factorization): See Section 39.7. 

DSTU: See diagonally scaled totally unimodular. 

ATP: See triangular totally positive. 

dual basis: A specific basis for the dual space defined from a given basis for the vector space. 3.8 

dual cone (of cone K ): See dual space of cone. 

dual norm (of vector norm || - ||): |ly||? = mMaxx40 =. 37.1 

dual space: The vector space of linear functionals on a given vector space. 3.8 

dual space (of cone K): K* = {ye V|(x,y) > 0 Vxe K}8.5, 26.1 


E 


ED: See Euclidean domain. 

ED-RCF basis (for a linear operator): See Section 6.4. 

ED-RCF matrix (over F ): A block diagonal matrix of the form C(h{") ® --- @ C(h;"") where each h;(x) 
is a monic polynomial that is irreducible over F . 6.4 

EDD: See elementary divisor domain. 

edge: A pair of vertices, part of a graph. 28.1 

edge cut: Let G = (V, E) bea graph, and let X, V \ X be a nontrivial partitioning of V. The set Ex of 
edges of G that have one end point in X and the other end point in V\ X is an edge cut. 36.2 

efficacy (of a processor): See speed. 

efficient: One algorithm is more efficient than another, if it accomplishes the same task with a lower cost 
of computation. 37.7 

eigenpair (of A € C”*"): A (eigenvalue, eigenvector) pair. 15.1 

eigenspace (of eigenvalue A of A): ker(A — AJ). 4.3 

k-eigenspace (of matrix or linear operator A at 1): ker(A — AI 6.1 
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eigentriplet (of A ¢ C”*”): A vector-scalar-vector triplet (y, 4, x) such that x 4 0, y 4 0, and Ax = ix, 
y’A=)dy*. 15.1 

eigenvalue (of matrix or linear operator A): A scalar A such that there exists a nonzero vector x such that 
Ax = ix. 4.3 

eigenvalue (of the pencil A — xB): 4 = w/v where vAv = wBvand v ¥ 0 (if v = 0,4 = 00). 43.1 
eigenvalue (of a graph): An eigenvalue of its adjacency matrix. 28.3 

eigenvalue decomposition (EVD) (of real symmetric matrix A): A= UA U', where U'U = UU! = I,, 
A = diag(A1,... , An) isa real diagonal matrix of eigenvalues, and the columns of U are eigenvectors. 42.1 
eigenvector (of matrix or linear operator A): A nonzero vector x such that there exists a scalar A such that 
Ax = ix. 4.3 

eigenvector (of the pencil A — xB): Nonzero vector v € C” such that there are scalars jz, v € C, not both 
zero, such that vAv = p Bv. 43.1 

elementary bidiagonal matrix: An n x n real matrix whose main diagonal entries are all equal to one, 
and there is at most one nonzero off-diagonal entry and this entry must occur on the super- or sub- 
diagonal. 21.2 

elementary divisor domain (EDD): A GCDD such that for any three elements a,b,c € D there exists 
P94, y € D such that (a,b,c) = (px)a + (py)b + (qy)c. 23.1 

elementary divisors (ofan ED-RCF matrix): The polynomials whose companion matrices are the diagonal 
blocks. 6.4 

elementary divisors (of matrix or linear operator A): The elementary divisors of RCFg p(A). 6.4 
elementary divisors rational canonical form (of matrix A € F"*"): An ED-RCF matrix that is similar 
to A. 6.4 

elementary divisors rational canonical form (of a linear operator): See Section 6.4. 

elementary divisors rational canonical form matrix: See ED-RCF matrix. 

elementary matrix: The result of doing one elementary row operation on the identity matrix. 1.5 
elementary column operation: Column analog of an elementary row operation. 6.5, 23.2 

elementary row operation (on a matrix over a domain): Add a multiple of one row to another, interchange 
two rows, or multiply a row by an invertible domain element. 6.5, 23.2 

elementary row operation (on a matrix over a field): Add a multiple of one row to another, interchange 
two rows, or multiply a row by a nonzero field element. 1.3 

elementary symmetric function: The sum of all products of n variables taken k at a time. Preliminaries 
elimination graph: A graph, digraph or bigraph associated with the principal submatrix that remains to 
be factored during Gaussian elimination. 40.4 

elimination ordering (for the rows of a matrix): A bijection that specifies the order in which the rows of 
the matrix are eliminated during Gaussian elimination. 40.5 

elimination sequence: See elimination ordering. 

embedding: A representation of a graph in R” where edges intersect only at vertices. 28.2 

empty graph: A graph with no edges. 28.1 

endpoint (of an edge in a graph): One of the two vertices on the edge. 28.1 

energy norm: See M-norm (alphabetized under norm). 

i, j-entry: The scalar in rowi and column j ina matrix. 1.2 

entry sign symmetric P- matrix A: A P-matrix such that for all i,j, either ajjaj; > 0 or ajj = 


ai; = 0.35.10 
entry sign symmetric Py- matrix A: A Po-matrix such that for all i,j, either ajjaj; > 0 or ajj = 
ji = 0. 35.10 
entry sign symmetric Po,,- matrix A: A Po -matrix such that for all i, j, either ajjaj; > 0 or ajj = 
Aji = 0. 35.10 


entry weakly sign symmetric P- matrix A: A P-matrix such that for all i, j, ajjaj; = 0. 35.10 
entry weakly sign symmetric Po- matrix A: A Po-matrix such that for all i, j, ajjaj; => 0.35.10 
entry weakly sign symmetric Pp,- matrix A: A Pp,-matrix such that for all i, j, ajjaj; = 0. 35.10 
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envelope (of sparse matrix A): The set of indices of the elements between the first and last nonzero elements 
of every row. 40.5 

equivalence relation: A relation that is reflexive, symmetric, and transitive. Preliminaries 

equivalent (bilinear forms f, g): There exists an ordered basis B such that the matrix of g relative to B is 
congruent to the matrix of f relative to B. 12.1 

g-equivalent (bilinear forms f, g): There exists an ordered basis B such that the matrix of g relative to B 
is p-congruent to the matrix of f relative to B. 12.4 

*equivalent (bilinear forms f,g on a complex vector space): g-equivalent with g = complex 
conjugation. 12.5 

equivalent (matrices A, B € F”*"): B = QAP for some invertible matrices Q, P. 2.4 

equivalent (matrices A, B € D”*"): B = QAP for some D-invertible matrices Q, P. 23.2 

equivalent: Systems of linear equations that have the same solution set. 1.4 

ergodic class (of stochastic matrix P ): A basic class of P. 9.4 

ergodic state: A state in an ergodic class. 9.4 

ergodicity coefficient (of complex matrix A): max{|A| : 4 € o(P) and |A| 4 p(A)}. 9.1 

error at step k of an iterative method for solving Ax = b: The difference between the true solution A~'b 
and the approximate solution x;,: eg = A~'b — x. 41.1 

Euclid’s Algorithm: See Section 23.1. 

Euclidean distance (matrix A): There exist vectors x|,...,X, € R? (for some d > 1) such that aij = 
|x; — x;ll2 for alli, j =1,...,n.35.3 

Euclidean domain (ED): A domain D with a function d : D\{0} — Z* such that for all nonzero a,b € 
D, d(a) < d(ab) and there exist t,r € D such that a = th+r, where either r = 0 or d(r) < d(b). 23.1 
Eulidean norm (of a matrix): See Frobenius norm. 

Eulidean norm (of a vector): See 2-norm. 

Euclidean space: Finite dimensional real vector space with standard inner product. 5.1 

EVD: See eigenvalue decomposition. 

even (permutation): Can be written as the product of an even number of transpositions. Preliminaries 
even (cycle in a sign pattern): The length of the simple or composite cycle is even. 33.1 

expanders (or family of expanders): An infinite family of graphs with constant degree and isoperimetric 
number bounded from below. 28.5 

explicit restarting (of the Arnoldi algorithm): A straightforward but inferior way to implement polyno- 
mial restarting by explicitly constructing the starting vector y(A)v by applying (A) through a sequence 
of matrix-vector products. 44.5 

exponent (of floating point number x): ¢ in floating point number x = + (;#;) b°. 37.6 

exponent (of a primitive digraph): Period; equivalently, the smallest value of k that works in the definition 
of primitivity. 29.6 

exponent (of a primitive matrix A): The exponent of the digraph of A. 29.6 

exponent of matrix multiplication complexity: The infimum of the real numbers w such that there exists 
an algorithm for multiplying n x n matrices with O(n") arithmetic operations. 47.2 

exponential (of a square complex matrix A): The matrix defined by the exponential power series, 
eA=ItAF A he tA pe 113 

extended precision: Floating point numbers that have greater than double precision. 37.6 

extension: Signing D’ = diag(d}, d},..., dj) isan extension of signing D if D’ # D and d/ = d; whenever 
d; # 0. 33.3 

external direct sum (of vector spaces): The cartesian product of the vector spaces with component-wise 
operations. 2.3 

exterior power (of a vector space): See Grassman space. 

exterior product (of vectors V),..., Vin)! Vi A+++ A Vn = mIAIt(v; ® --- @ vz). 13.6 

extreme point (of a closed convex set S): A point in S that is not a nontrivial convex combination of other 
points in S. Preliminaries 

extreme vector v € K: Either v is the zero vector or v is nonzero and ®(v) = {Av: A > 0}. 26.1 
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face F (of cone K): A subcone of K such that v € F, v>* w>* 0=5 we F. 26.1 

face of K generated by S C K: The intersection of all faces of K containing S. 26.1 

family of Toeplitz matrices (defined by symbola = )° po axz*): The set { Tn} n>1 where T,, has constants 
Ay—ny+ ++ Any. 16.2 

Fiedler vector (of simple graph G): An eigenvector of the Laplacian of G corresponding to the algebraic 
connectivity. 36.1 

field: A nonempty set with two binary operations, addition and multiplication, satisfying commutativity, 
associativity, distributivity and existence of identities and inverses. Preliminaries 

field of rational functions: The quotient field of the field of polynomials over F. 23.1 

field of values: See numerical range. 

fill element (in a sparse matrix A): An element that is zero in A but becomes nonzero during Gaussian 
elimination. 40.3 

fill graph (of sparse matrix A): The (graph, digraph, or bigraph) of A, together with all the additional 
edges corresponding to the fill elements that occur during Gaussian elimination. 30.2, 40.5 

fill matrix (of Gaussian elimination ona matrix): The matrix M,+.M)+---+M,_,;—(n—1)I+U, where the 
M; are Gauss transformations used in the elimination and U is the resulting upper triangular matrix. 40.3 
final (subset a of vertices): no vertex in a has access to a vertex not in a. 9.1 

finite dimensional (vector space V): V has a basis containing a finite number of vectors. 2.2 

finitely generated (ideal J of a domain D): An ideal of D that is finitely generated as a D-module. 23.1 
finitely generated (module M over a domain D): There exist k elements (generators) v,,... , VK € M so 
that every v € Misa linear combination of v;,... , vz, over D. 23.3 

fixed space: the set of vectors fixed by a linear transformation. 3.6 

flip map: See Section 22.2. 

floating point addition, subtraction multiplication, division: Operations on floating point numbers, 
See Section 37.6. 

floating point number: A real number of the form x = + (;7;) b°. 37.6 

floating point operation: One of floating point addition, subtraction, multiplication, division. 37.6 
flop: A floating point operation. 37.7 

FOM: See full orthogonalization method. 

forest: A graph with no cycles. 28.1 

forward error (in f (x) ): f(x) -— F(x), the difference between the mathematically exact function evaluation 
and the perturbed function evaluation. 37.8 

forward stable (algorithm): The forward relative error is small for all valid input data x despite rounding 
and truncation errors in the algorithm. 37.8 

free variable: A variable x; in the solution to a system of linear equations that is a free parameter. 1.4 


Frobenius norm (of m x n complex matrix A): || Allp = ,/>°j2, vi=1 |a;;|*. 7.1, 37.3 


Frobenius normal form (of matrix A): A block upper triangular matrix with irreducible diagonal blocks 
that is permutation similar to A. 27.3 

full (cone): Has a nonempty interior. 8.5, 26.1 

fully indecomposable ((0, 1)-matrix): Not partly decomposable. 27.2 

full orthogonalization method (FOM): Generates, at each step k, the approximation x; of the form 
X, € Xo + Span{ro, Aro,... , A*~!x9} for which the residual is orthogonal to the Krylov space. 41.3 
fundamental subspaces (of the least squares problem Ax = b): rangeA, ker A?, ker A and range A’. 39.3 


—0o 


G 


Galerkin condition: The defining condition for a Ritz vector and Ritz value. 44.1 
Gauss multipliers: The nonzero entries in a Gauss vector. 38.3 
Gauss transformation: The matrix M, = I — exe , where £; is a Gauss vector. 38.3 
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Gauss vector: A vector £; € C” with the leading k entries equal to zero. 38.3 

Gauss-Jordan elimination: A process used to find the reduced row echelon form of a matrix. 1.3 
Gaussian elimination: A process used to find a row echelon form of a matrix. 1.3, 38.3 

g.c.d.: See greatest common divisor. 

GCDD: See greatest common divisor domain. 

general linear group (of order n over F): The multiplicative group consisting of all invertible n x n 
matrices over the field F . 67.1 

general solution: A formula that describes every vector in the solution set to a system oflinear equations. 1.4 
generalized cycle (in a digraph): A disjoint union of one or more cycles such that every vertex lies on 
exactly one cycle. 29.1 

generalized cycle product: A walk product where the walk is a generalized cycle. 29.4 

generalized diagonal (of square matrix A): Entries of A corresponding to a generalized cycle in the digraph 
of A. 29.4 

generalized eigenspace (of matrix or linear operator A at A): ker(A—A)”, where v is the index of AatA.6.1 
generalized eigenvector (ofa matrix or linear operator): A nonzero vector in the generalized eigenspace. 6.1 
generalized Laplacian (of simple graph G): A matrix M such that fori # j, mj; < 0 ifthe ith and jth 
vertices are adjacent and mj; = 0 otherwise (no restriction fori = j). 28.4 

generalized line graph: See Section 28.2. 

generalized minimal residual (GMRES): Algorithm generates, at each step k, the approximation x; of the 
form x; € Xp + Span{ro, Aro,... , A*~!19} for which the 2-norm of the residual is minimal. 41.3 
generalized sign pattern: A matrix whose entries are from the set {+,—,0,#}, where # indicates an 
ambiguous sum (the result of adding + with —). 33.1 

generalized star: A tree in which at most one vertex has degree greater than two. 34.6 

generators (of an ideal): See finitely generated. 

generic matrix: A matrix whose nonzero elements are independent indeterminates over the field F . 30.2 
geometric multiplicity: The dimension of the eigenspace. 4.3 

GerSsgorin discs (of A € C”*"):{z EC : |z—aji| < ee |a;j|} fori = 1,...,n. 14.2 

Givens matrix: See Givens transformation. 

Givens rotation: See Givens transformation. 

Givens transformation: An identity matrix modified so that the (i,7) and (j, j) entries are replaced 
by c = cos(@), the (i, j) entry is replaced by s = e'” sin(@), and the (j,i) entry is replaced by —5 = 
—e'” sin(6). 38.4 

GMRES: See generalized minimal residual, restarted GMRES algorithm. 

graded (associative algebra A): There exist (Ax)zen vector subspaces of A such that A = @yzen Ax and 
Aj Aj Cc Aj+j for every 1, j EN. 13.9 

graded matrix A € C”*": A can be scaled as A = GS (or A = S*GS, depending on situation) such that 
G is “well-behaved” (i.e., k2(G) is of modest magnitude), where S is a scaling matrix (often diagonal). 15.3 
Gram matrix (of vectors vj,...,V, in a complex inner product space): the matrix whose i, j-entry is 
(vj Vj ) . 8.1 

graph: A finite set of vertices and a finite multiset of edges, where each edge is an unordered pair of vertices 
(not necessarily distinct). 28.1 

graph (of n x n matrix A): The simple graph whose vertex set is {1,... ,} and having an edge between 
vertices i and j (i # j) ifand only ifa;; 4 0 or aj; A 0. 19.3, 34.1 

graph (of a convex polytope P): Vertices are the extreme points of P and edges are the pairs of extreme 
points of 1-dimensional faces of P. 27.6 


graph (of n x n combinatorially symmetric partial matrix B): The graph having vertices V = {1,...,”} 
and for eachi, j € V, {i, j} isan edge exactly when b;; is specified (loops permitted, no multiple edges). 35.1 
graph ofacombinatorially symmetric n x n sign pattern A = [a;;]: The graph with vertex set {1,2,..., 1} 


where {i, j} is an edge iff a;; A 0 (note: the graph may have loops). 33.5 


Grassman algebra (on vector space V): /\ V= ‘<3 ( A‘ v) with alt multiplication. 13.9 
keN 
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Grassmann space (of vector space V): A” V =Alt(®)” V). 13.6 

greatest common divisor (g.c.d.) (of a),...,4, € D): d € D such that dla; fori = 1,...,n, and if 
d'\a;,i = 1,...,n, then d’|d. 23.1 

greatest common divisor domain (GCDD): A domain in which any two elements have a g.c.d. 23.1 
greatest integer (of a real number x): the greatest integer less than or equal to x. Preliminaries 

group: A set with one binary operation, satisfying associativity, existence ofidentity and inverses.Preliminaries 
group inverse (of square complex matrix A): A matrix X satisfying AXA = A, XAX = X and 
AX = XA.9.1 


H 


H-matrix (real square matrix A): The comparison matrix of A is an M-matrix. 19.5 

Hadamard matrix: A +1-matrix H,, with H,, HI = nl, 32.2 

Hadamard product (of A, B € F"*"): The n x n matrix whose j, j-entry is a;jb;;. 8.5 

Hall matrix: An n x n (0, 1)-matrix that does not have a p x q zero submatrix for positive integers p,q 
with p+q > n. 27.2 

Hamilton cycle: A cycle in a graph that includes all vertices. 28.1 

Hamiltonian cycle: See Hamilton cycle. 

Hankel matrix: A matrix with constant elements along its antidiagonals. 48.1 

height (of basic class B): The largest number of vertices on a simple walk that ends at B in the basic 
reduced digraph. 9.3 

hereditary (class of matrices X): Whenever A is an X-matrix anda@ C {1,...,n}, then A[q] is an 
X-matrix. 35.1 

Hermite normal form: A generalization of reduced row echelon form used in domains. 23.2 

Hermitian (linear operator on an inner product space): An operator that is equal to its adjoint. 5.3 
Hermitian (matrix): A real or complex matrix equal to its conjugate transpose. 1.2, 7.2, 8.1 

Hermitian adjoint (of a matrix): The conjugate transpose of a real or complex matrix. 1.2 

Hermitian form f (on complex vector space V): A sesquilinear form such that f(v,u) = f (u,v) for all 
u,v € V. 12.5 

Hoffman polynomial (of graph G): A polynomial h(x) of minimum degree such that h(Ag) = J. 28.3 
Holder norm: See p-norm. 

homogeneous (digraph): Either symmetric or asymmetric. 35.7 

homogeneous (element of graded algebra A = @jc<n Ax): An element of some Ax. 13.9 

homogeneous (pencil associated with A(x) = Ap + xA; € D[x]”™*"): A(x, x1) = xoAo + x1 Al € 
D[xo, x |"*". 23.4 

homogeneous (polynomial): All terms have the same degree. 23.1 

homogeneous (system of linear equations): A system in which all the constants are zero. 1.4 
Householder matrix: See Householder transformation. 

Householder reflector: See Householder transformation. 

Householder transformation (defined by v € C”): The matrix I — Trae 38.4 

Householder vector: The vector v used to define a Householder transformation. 38.4 

hyperbolic SVD decomposition (of matrix pair (G, 7)): A decomposition of G,G = WX B7!, where W 
is orthogonal, & is diagonal, and B is 7—orthogonal. 46.5 


I 


IAP: See inertially arbitrary pattern. 

ideal (ofa domain D): A nonempty subset I of D such that ifa, b € I and p,q € D, then pa+qb € 1.23.1 
idempotent: A matrix or linear transformation that squares to itself. 2.7, 3.6 

identity matrix: A diagonal matrix having all diagonal elements equal to one. 1.2 

identity pattern (of order n): The n x n diagonal pattern with + diagonal entries. 33.1 

identity transformation: A linear operator that maps each vector to itself. 3.1 
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IEP: See inverse eigenvalue problem. 

IF-RCF matrix: A block diagonal matrix of the form C(a,) ®@ --- 6 C(a,), where a;(x) divides a;+)(x) 
fori =1,...,s — 1.6.6 

ill-conditioned (input z for P): Some small relative perturbation of z causes a large relative perturbation 
of P(z). 37.4 

image: See range. 

imaginary part (of complex number a + bi): b. Preliminaries 

immanant: A function f, from complex square matrices to complex numbers defined by the irreducible 
character of a partition A of n, specifically, f,(M) = ses: X.(0) TTL, Mio(i)- 31-10 

implicit restarting (of the Arnoldi algorithm): Apply a sequence of implicitly shifted QR steps to an m-step 
Arnoldi or Lanczos factorization to obtain a truncated form of the implicitly shifted QR-iteration. 44.5 
imprimitive (digraph): Not primitive. 29.6 

imprimitive (matrix A): The digraph '(A) is imprimitive. 29.6 

improper (divisor of a in a domain): An associate of a or a unit. 23.1 

incidence matrix (of a graph): A matrix with rows indexed by the vertices and columns indexed by the 
edges, having i, j entry 1 if vertex v; is an endpoint of edge e; and 0 otherwise. 28.4 

incidence matrix (of subsets S,, S:,...,S,, of a finite set {x,, x2,...,x,}): The m x n (0,1)-matrix 
M= [mij] in which mij = 1 iff x; € §;. 31.3 

(+1)-incidence matrix (of a 2-design): A matrix W whose rows are indexed by the elements x; of X and 
whose columns are indexed by the blocks B; having w;; = —1 if x; € B; and w;; = +1 otherwise. 32.2 
incomplete Cholesky decomposition: A preconditioner for a Hermitian positive definite matrix A of the 
form M = LL*, where L is a sparse lower triangular matrix. 41.4 

incomplete L U decomposition: A preconditioner for a general matrix A of the form M = LU, where L 
and U are sparse lower and upper triangular matrices. 41.4 

inconsistent: A system of linear equations that has no solution. 1.4 

indefinite (Hermitian matrix A): Neither A nor — A is positive semidefinite. 8.4 

independent (subspaces W;): w) + --- + we = 0 and w; € W,,i = 1,...,k implies w; = 0 for all 
i=1,...,k.2.3 

independent set of vertices: See coclique. 

index (of an eigenvalue): The smallest integer k such that the k-eigenspace equals the k + 1-eigenspace. 6.1 
index (of square nonnegative matrix P ): The index of the spectral radius of P. 9.3 

index (of A € Ho): See Section 23.3. 

index of imprimitivity (of an irreducible matrix A): The greatest common divisor of the lengths of all 
cycles in the digraph (A); same as period. 29.7 

index of primitivity: See exponent. 


individual condition number for eigenvalue A: eee where (y, A, x) is an eigentriplet. 15.1 
induced basis: The nonzero images in the quotient V/W of vectors in a basis that contains a basis for W. 2.3 
induced norm: The matrix norm induced by the family of vector norms || - || is || Al] = max. cl . 37.3 


induced subdigraph (of [ = (V,E£)): A subdigraph T = (V’, E’) containing all arcs from E with 
endpoints in V’. 29.1 

induced subgraph (of G = (V, E)): Asubgraph G = (V’, E’) containing all edges from E with endpoints 
in V’. 28.1 

inertia (of complex square matrix A): The ordered triple in(A) = (7 (A), v(A), 6(A)) where (A) is the 
number of eigenvalues of A with positive real part, v(A) is the number of eigenvalues of A with negative 
real part, and 6(A) is the number of eigenvalues of A on the imaginary axis. 8.3, 19.1 

inertia preserving (real square matrix A): The inertia of AD is equal to the inertia of D for every 
nonsingular real diagonal matrix D. 19.3 

inertially arbitrary pattern (IAP): A sign pattern A of size n such that every possible ordered triple 
(p,q,Z) of nonnegative integers p,q, and z with p + q + z = ncan be achieved as the inertia of some 
Be Q(A). 33.6 

infinite dimensional (vector space): A vector space that is not finite dimensional. 2.2 
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initial (minor): See Section 21.3 

injective: See one-to-one. 

inner distribution: Vector parameter of an association scheme. 28.6 

inner product (on a vector space V over F = Ror C): A function (-,-): V x V > F satisfying certain 
conditions. For F = R, (-,-) is symmetric, bilinear and positive definite. 5.1 

inner product space: A real or complex vector space with an inner product. 5.1 

integral domain: A commutative ring without zero divisors and containing identity 1. 23.1 

iterative method (for solving a linear system Ax = b): Any algorithm that starts with an initial guess x9 for 
the solution and successively modifies that guess in an attempt to obtain improved approximate solutions 
X1,X2,.... 41.1 

internal direct sum: See direct sum. 

intersection (of two graphs G = (V, E) and G’ = (V’, E’)): The graph with vertex set VM V’, and edge 
(multi)set E 9 E’. 28.1 

intersection numbers: Parameters of an association scheme. 28.6 

P-invariant face (of K -nonnegative matrix P): PF C F.26.1 

invariant factors (of IF-RCF matrix C(a,) ® --- ®@ C(a,)): The polynomials a;(x),i = 1,...5. 6.6 
invariant factors (of matrix or linear operator A € F"*"): The invariant factors of the invariant factors 
rational canonical form of A. 6.6 

invariant factors (of a matrix over a domain): See Section 23.1. 

invariant factors rational canonical form (of matrix A €¢ F"*"): The IF-RCF matrix similar to A. 6.6 
invariant factors rational canonical form (of a linear operator): See Section 6.6. 

invariant factors rational canonical form matrix: See [F-RCF matrix. 

invariant polynomials: See invariant factors. 

invariant subspace (of linear transformation T : V — W): A subspace U of V such that for all u € U, 
T(u) € U.3.6 

inverse (of square matrix A): A matrix B such that AB = BA = I.1.5 

inverse (of linear transformation T : V — W): A linear transformation S : W — V such that TS = Iw 
and ST = Iy.3.7 

inverse eigenvalue problem: The problem of constructing a matrix with prescribed structural and spectral 
constraints. 20.1 

inverse eigenvalue problem with prescribed entries: Construct a matrix with given eigenvalues subject 
to given entries in given positions. 20.1 

Inverse Eigenvalue Problem of tree T: Determine all possible spectra that occur among matrices in 
S(T). 34.5 

inverse iteration: The power method applied to the inverse of a shifted matrix. 42.1 

inverse M-matrix: An invertible matrix whose inverse is an M-matrix. 9.5 

inverse nonnegative: A square sign pattern that allows an entrywise nonnegative inverse. 33.7 
inverse-positive (real square matrix A): A is nonsingular and A~! > 0.9.5 

inverse positive (square sign pattern): Allows an entrywise positive inverse. 33.7 

invertible (matrix or linear transformation): Has an inverse. 1.5, 3.7 

D-invertible (matrix in D”*"): Has an inverse within D”*". 23.2 

irreducible (element a of a domain): a is not a unit and every divisor of a is an associate of a or a unit. 23.2 
irreducible (matrix): Not reducible. 9.2, 27.3 

K-irreducible (K -nonnegative matrix P ): The only P-invariant faces are the trivial faces. 26.1 
irreducible components (of matrix A): The diagonal blocks in the Frobenius normal form of A. 27.3 
isolated vertex: A vertex of a graph having no incident edges. 28.1 

isomorphic (graphs G = (V, E) and G’ = (V’, E’)): There exist bijections @ : V > V’andy: E > E’, 
such that v € V is an endpoint of e € E ifand only if #(v) is an endpoint of w(e). 28.1 

isomorphic (vector spaces): There is an isomorphism from one vector space onto the other. 3.7 
isomorphism (of vector spaces): An invertible linear transformation. 3.7 

isoperimetric number: A parameter of a simple graph. 28.5 
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J-orthogonal: Alphabetized under orthogonal. 

Jacobi method (for computing the EVD of A): A sequence of matrices, Ay = A, Agy1 = G(tk; jes 55) 
AxG (tks Jk c,s)', k= 1,2,... , where G(iz, jk» €,S) is a Givens rotation matrix. 42.7 

Jacobi rotation: A Givens rotation used in the Jacobi method. 42.7 

join (of disjoint graphs G = (V, E) and G’ = (V’, E’): The union of G U G’ and the complete bipartite 
graph with vertex set V U V’ and partition {V, V’}. 28.1 

join (of faces F, G of cone K ): Face generated by F U G. 26.1 

Jordan basis (for matrix A): The ordered set of columns of C where JCF(A) = C7! AC. 6.2 

Jordan basis (of a linear operator): See Section 6.2. 

Jordan block (of size k with eigenvalue 1): The k x k matrix having every diagonal entry equal to A, every 
first superdiagonal entry equal to 1, and every other entry equal to 0. 6.2 

Jordan canonical form (of matrix A): A Jordan matrix that is similar to A. 6.2 

Jordan canonical form (of a linear operator T): g[T], that is a Jordan matrix. 6.2 

Jordan chain (above eigenvector x for eigenvalue A of A): A sequence of vectors Xp = xX, X),..., X, Satisfying 
xj = (A—AI)xj4, fori = 0,...,h — 1 (where h is the depth of x). 6.2 

Jordan invariants (of matrix or linear operator A): The set of distinct eigenvalues of A and for each eigen- 
value A, the number by and sizes pi,..., pp, of the Jordan blocks with eigenvalue 1 in a Jordan canonical 
form of A. 6.2 

Jordan matrix: A block diagonal matrix having Jordan blocks as the diagonal blocks. 6.2 


K 


K-irreducible, K -nonnegative, K -positive, K-semipositive: Alphabetized under irreducible, nonnega- 
tive, positive, semipositive. 

ker: See kernel. 

kernel (ker) (of a linear transformation): The set of vectors mapped to zero by the transformation. 3.5 
kernel (ker) (of a matrix A): The set of solutions to Ax = 0. 2.4 

Kronecker product (of matrices A and B): The block matrix whose i, j block is a;; B 10.4 

Krylov matrix (of matrix A, vector x, and positive integer k): [x, Ax, A2x,--- , AK—!x]. 42.8 

Krylov space: A subspace of the form Span{q, Aq, A*q,... , A‘~!q}, where A is an n by n matrix and q is 
an n-vector. 41.1, 44.1 

Ky Fan k norms (of A € C’”*"): || Allg = So) 0;(A). 17.3 


L 


L-matrix (sign pattern or real matrix A): For every B € Q(A), the rows of B are linearly independent. 33.3 
Lanczos algorithm (for Hermitian matrices): A short recurrence for constructing an orthonormal basis 
for a Krylov space. 41.2 

Laplacian: See Laplacian matrix. 

Laplacian eigenvalues (of simple graph G): The eigenvalues of the Laplacian matrix of G. 28.4 
Laplacian matrix (of simple graph G): The matrix D — Ag, where D is the diagonal matrix of vertex 
degrees and Ag is the adjacency matrix of G. 28.4 

Laplacian matrix (of a weighted graph): The matrix L = [¢;;] such that ¢;; = 0 if vertices i and j are 
distinct and not adjacent, ¢;; = —w(e) ife = {i, j} is an edge, and €;; = }° w(e), where the sum is taken 
over all edges e incident with vertex i. 36.4 

largest size of a zero submatrix: The maximum of r + s such that the matrix has an r x s (possibly 
vacuous) zero submatrix. 27.1 

L DU factorization: A factorization of a matrix as the product of a unit lower triangular matrix, a diagonal 
matrix, and a unit upper triangular matrix. 1.6 

leading principal minor: The determinant of a leading principal submatrix. 4.2 
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leading principal submatrix: A principal submatrix lying in rows and columns | to k. 1.2 

leading entry: The first nonzero entry in a row of a matrix in REF. 1.3 

least squares data fitting: See Section 39.2. 

least squares problem (Ax = b): Find a vector x € R” such that ||b — Ax||, is minimized. 5.8, 39.1 

least squares solution: A solution to a least squares problem. 5.8, 39.1 

left singular space: See singular spaces. 

left singular vector: See singular vectors. 

left eigenvector (of matrix A): A nonzero row vector y such that there exists a scalar ) such that yA = 
dy. 4.3 

length (of a walk, path, cycle in a graph or digraph): The number of edges (arcs) in the walk. 28.1, 29.1 
length (of a permutation cycle): The number of elements in the cycle. Preliminaries 

length (of a vector): See norm. 

(C, 0)-limit (of complex sequence {4,}:n=0,1,...): Ordinary limit limy,,o0 @m. 9.1 

(C, 1)-limit (of complex sequence {4m} n=0,1,...): iMm—+oo mm} DAEs a;.9.1 

(C, k)-limit (of complex sequence {4} m—0,1,...): Defined inductively from (C, k — 1)-limit. See Section 9.1. 
line (of a matrix): A row or column. 27.1 

line graph (of simple graph G): The simple graph that has as vertices the edges of G, and vertices are 
adjacent if the corresponding edges of G have an endpoint in common. 28.2 

linear combination: A (finite) sum of scalar multiples of vectors. 2.1 

linear equation: An equation of the form a,x; +--+ + a,x, = b.1.4 

linear form: See linear functional. 

linear functional: A linear transformation from a vector space to the field. 3.8 

m-linear map: See multilinear map. 

linear mapping: See linear transformation. 

linear operator: A linear transformation from a vector space to itself. 3.1 

linear preserver (of function f): A linear operator @ : V — V such that f(@(A)) = f(A) for every 
A € V, where V is a subspace of F™*". 22.1 

linear preserver problem (corresponding to the property P): The problem of characterizing all linear 
(bijective) maps on a subspace of matrices satisfying P. 22.2 

linear transformation: A function from a vector space to a vector space that preserves addition and scalar 
multiplication. 3.1 

linear transformation associated to matrix A: The transformation that multiplies each vector in F” by 
A. 3.3 

linearly dependent (set of vectors): There is a linear combination of the vectors with at least one nonzero 
coefficient, that is equal to the zero vector. 2.1 

linearly independent (set of vectors): Not linearly dependent. 2.1 

linked: Two disjoint Jordan curves in IR? such that there is no topological 2-sphere in R? separating 
them. 28.2 

linklessly embeddable: There is an embedding of graph G in R® such that no two disjoint cycles of G are 
linked. 28.2 

little-oh: Function f is o(g) if the limit of (4 is 0. Preliminaries 

local similarity: A map @ : F"*" — F"*" such that for every A € F"*" there exists an invertible 
Ra € F"*" such that (A) = RaAR{!. 22.4 

Loewner (partial) ordering (on Hermitian matrices A, B): A > B if A— B is positive definiteand A > B 
if A — B is positive semidefinite. 8.5 

logarithm (of a square complex matrix A): Any matrix B such that e? = A. 11.4 

look-ahead strategy: A technique to overcome breakdown in some algorithms. 41.3 

loop: An edge (arc) of a graph (directed graph) having both endpoints equal. 29.1 

Lovasz parameter: A parameter of a simple graph. 28.5 

lower Hessenberg (matrix A): A? is upper Hessenberg. 10.2 

lower shift matrix: A square matrix with ones on the first subdiagonal and zero elsewhere. 48.1 
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lower triangular matrix: A matrix such that every entry having row number less than column number is 
zero. 1.2, 10.2 

Léwner (partial) ordering: See Loewner (partial) ordering 

LU factorization: A factorization of a matrix as the product of a unit lower triangular matrix and an 
upper triangular matrix. 1.6, 38.3 

Lyapunov diagonally semistable (real square matrix A): There exists a positive diagonal matrix D such 
that AD + DA’ is positive semidefinite. 19.5 

Lyapunov diagonally stable (real square matrix A): There exists a positive diagonal matrix D such that 
AD + DA’ is positive definite. 19.5 

Lyapunov scaling factor (of real square matrix A): A positive diagonal matrix D such that AD + DAT is 
positive (semi)definite. 19.5 


M 


M-matrix (real square matrix A): A can be written as A = sI — P with P a nonnegative matrix and s a 
scalar satisfying s > p(P).9.5 

Mo-matrix: (real square matrix A): A can be written as A = sI — P with P a nonnegative matrix and s 

a scalar satisfying s > p(P). 9.5 

machine epsilon: The distance between the number one and the next larger floating point number. 37.6 

main angles (of a graph): The cosines of the angles between the eigenspaces of the adjacency matrix and 
the all-ones vector. 28.3 

main diagonal (of a matrix): The set of diagonal entries of a matrix. 1.2 

majorizes: Weakly majorizes and the sums of all entries are equal. Preliminaries 

(k-)matching (of a graph): A set of k mutually disjoint edges. 30.1 

matrices: Plural of matrix. 

matrix: a rectangular array of elements of a field. 1.2 

Or if specifically stated, a rectangular array of elements of a domain, ring, or algebra. The following 
standard matrix terms are applied to the matrices over a domain D viewed as in the quotient field of 
D: determinant, minor, rank, Section 23.2. The following standard matrix terms are applied to matrices 
over a Bezout domain within the domain: kernel, range, system of linear equations, coefficient matrix, 
augmented matrix, Section 23.3 

matrix condition number: See condition number (of matrix A for linear systems). 

matrix cosine: See cosine. 

matrix direct sum: A block diagonal matrix. 2.4, 10.2 

matrix exponential: See exponential. 

matrix function: A function of a square complex matrix; can be defined using power series, Jordan 
Canonical Form, polynomial interpretation, Cauchy integral, etc. 11.1 

matrix logarithm: See logarithm. 

matrix of a transformation T (with respect to bases 6 and C): The matrix consisting of the coordinate 
vectors with respect to C of the images under T of the vectors in B. 3.3 

matrix norm || - ||: A family of real-valued functions defined on m x n real or complex matrices for 
all positive integers m and n, such that for all matrices A and B (where A and B are compatible for 
the given operation) and all scalars, (1) || Al] > 0, and || Al] = 0 implies A = 0; (2) ||a@A|| = |a||| All; 
(3) |A+ Bll < |All + Bll; (4) ABI < |All Bll. 37.3 

matrix pencil: See pencil. 

matrix product: The result of multiplying two matrices. 1.2 

matrix representing f relative to basis B = (wi, W2,... , Wn): The matrix whose i, j-entry is f (wi, Wj), 
Section 12.1; also applied to y-sesquilinear form. 12.4 

matrix sine: See sine. 

matrix sign function: See sign (of a matrix). 

matrix square root: See square root, principal square root. 
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matrix—vector product: The result of multiplying a matrix and a vector. 1.2 

maximal (ideal I of a domain D): The only ideals that contain I are I and D, and I # D. 23.1 

maximal (sign-nonsingular-sign pattern matrix): If a zero entry is set nonzero, then the resulting pattern 
is not SNS. 33.2 

maximal rank (of sign-pattern A): max{rankB : B € Q(A) }. 33.6 

maximum column sum norm (of matrix A € C”*"): || All) = maxy<j<n 0/2, |aij|. 37.3 

maximum multiplicity (of simple graph G): The maximum multiplicity of any eigenvalue among matrices 
in S(G). 34.2 

maximum row sum norm (of matrix A € C”*"): || Allo = Maxi<i<m i= |a;;|. 37.3 

max-plus semiring: The set R U {—oo}, equipped with the addition (a,b) +» max(a, b) and the multi- 
plication (a, b) +> a+b. The identity element for the addition, zero, is —oo, and the identity element for 
the multiplication, unit, is 0. 25.1 

measure of relative separation (of positve real numbers a, b): |,/a/b — /b/a|. 17.4 

meet (of faces F, G of cone K): F 1 G 26.1 

metric: A distance function. Preliminaries 

MIEP: See multiplicative inverse eigenvalue problem. 

minimal (matrix norm || - ||): For any matrix norm || - ||,, || All) < || Al] for all _A implies || - ||, = || - ||. 37.3 
minimal polynomial (of matrix A): The unique monic polynomial of least degree for which q 4(A) = 0.4.3 
minimal rank (of sign pattern A): min {rankB : B € Q(A) }. 33.6 

minimal residual (MINRES) algorithm (to solve Hx = b with H preconditioned Hermitian): At each 
step k, the approximation x; is of the form x, € Xp + Span{r, Hr,... , H*~'19} for which the 2-norm 
of the residual, ||r,||2, is minimal. 41.2 

minimal sign-central: A sign-central pattern that is not sign-central if any column of A is deleted. 33.11 
minimally connected (digraph): Strongly connected and the deletion of any arc produces a subdigraph 
that is not strongly connected. 29.8 

minimally potentially stable (sign pattern A): A is a potentially stable, irreducible sign pattern such that 
replacing any nonzero entry by zero results in a pattern that is not potentially stable. 33.4 

minimum co-cover (of a (0, 1)-matrix): A co-cover with the smallest number of 1s. 27.1 

minimum cover (of a (0, 1)-matrix): A cover with the smallest number of lines. 27.1 

minimum-norm least squares solution: The least squares solution of minimum Euclidean norm. 
39.1 

minimum rank (of simple graph G): The minimum of rank A among matrices A € S(G). 34.2 

minor (of a graph G): Any graph that can be obtained from G by a sequence of edge deletions, vertex 
deletions, and contractions. 28.2 

minor (of a matrix): The determinant of a submatrix; for the i, j-minor, the submatrix is obtained by 
deleting row i and column j. 4.1 

MINRES: See minimal residual. 

Mobius function: pz : N+ {—1,0, 1} is defined by (1) = 1, w(m) = (—1)° ifm isa product of e distinct 
primes, and j.(m) = 0 otherwise. 20.5 

D-module: A generalization of vector space over a field; an additive group with a (scalar) multiplication by 
elements a € D that satisfies the standard distribution properties and lv = v for all v € D. The following 
standard terms are also applied to modules over a Bezout domain—linear combination, basis (may not 
exist), standard basis for D”, dimension (if there is a basis). 23.3 

monic (polynomial p(x)): The coefficient of the highest power of x in p(x) is 1. 23.1 

monotone (real vector v = [v;]): v) > v2 > +++ > Vy. 27.4 

monotone (vector norm || - ||): For all x, y, |x| < |y| implies ||x|| < |ly|]. 37.1 

Moore—Penrose pseudo-inverse (of matrix A € C”*"): A matrix A’ € C”*™ satisfying: AATA = 
A; A'AAT= Al; (AAT)* = AAT; (ATA)* = ATA.5,7 

Moore-Penrose inverse (of a sign pattern): Analog of the Moore—Penrose inverse of a real matrix. 33.7 
multibigraph (of a nonnegative integer matrix A): Like the bigraph of A, except that edge {i, j’} has 
multiplicity a;;. 30.2 
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multigrid preconditioner: A preconditioner designed for problems arising from partial differential equa- 
tions discretized on grids. 41.4 

multilinear form: A multilinear map into the field F. 13.1 

multilinear map (m-linear map): A map ¢ from V, x --- x V,, into U that is linear on each coordinate. 13.1 
multiple (eigenvalue A of real symmetric matrix B): ag(A) > 1.34.1 

multiplicative (map @ : F"*" > F"*"): 6(AB) = @(A)@(B), for all A, B € F"*". 22.4 

multiplicative D-stable (real square matrix A): DA is stable for every positive diagonal matrix D. 19.3 
multiplicative inverse eigenvalue problem (MIEP): Given B € C"*" and Aj,... 5 An € C find a diagonal 
matrix D € C"*" such that 0(BD) = {Aj,... 5 An}. 20.9 

multiplicative perturbation (of A ¢ C’”*”): Dj ADp for some D; € C™™*", Dr € C"*". 15.3 
multiplicative preserver: A multiplicative map preserving a certain property. 22.4 

multiplicity (of an eigenvalue): See algebraic multiplicity, geometric multiplicity. 

multiplicity (of a singular value): The dimension of the right or left singular space. 45.1 

multiset: An unordered list of elements that allows repetition. Preliminaries 


N 


NaN: Result of an operation that is not a number in a standard-conforming floating point system. 37.6 
nearly decomposable ((0, 1)-matrix): Fully indecomposable and each matrix obtained by replacing a 1 
with a 0 is partly decomposable. 27.2 

nearly reducible (matrix A): Irreducible and each matrix obtained from A by replacing a nonzero entry 
with a zero is reducible. 27.3 

nearly sign-central: A sign pattern that is not sign-central, but can be augmented to a sign-central pattern 
by adjoining a column. 33.11 

nearly sign nonsingular (NSNS) (sign pattern): A square pattern having at least two nonzero terms in the 
expansion of its determinant, with precisely one nonzero term having opposite sign to the others. 33.2 
negative definite (real symmetric or Hermitian bilinear form f): — f is positive definite. 12.2, 12.5 
negative semidefinite (real symmetric or Hermitian bilinear form f ): — f is positive semidefinite. 12.2, 12.5 
negative set of edges a (in a signed bipartite graph): sgn(@) is —1. 30.1 

negative stable (complex polynomial): Its roots lie in the open left half plane. 19.2 

negative stable (complex square matrix): Its eigenvalues lie in the open left half plane. 19.2 

neighbor (of vertex v): Any vertex adjacent to v. 28.1 

NIEP: See nonnegative inverse eigenvalue problem. 

nilpotent: A matrix or linear transformation that can be multiplied by itself a finite number of times to 
obtain the zero matrix or transformation. 2.7, 3.6 

node: See vertex. 

noncommutative algorithms: Algorithms that do not use commutativity of multiplication. 47.2 
nondefective (matrix A € F”"*"): For each eigenvalue of A (over algebraic closure of F ), the geometric 
multiplicity equals the algebraic multiplicity. 4.3 

nondegenerate (bilinear form f on vector space V): The rank of f is equal to dim V, Section 12.1; also 
applied to y-sesquilinear form. 12.4 

nonderogatory (matrix A € F"*"): The geometric multiplicity of each eigenvalue of A (over algebraic 
closure of F) is 1. 4.3 

nondifferentiable (boundary point jz of convex set S): There is more than one support line of S passing 
through jz. 18.2 

non-Hermitian Lanczos algorithm See Section 41.3 

nonnegative (real matrix A): All of A’s elements are nonnegative. 9.1 

nonnegative (sign pattern): All of its entries are nonnegative. 33.7 

K-nonnegative (matrix A € R’*"): AK C K. 26.1 

K-nonnegative (vector v € R"):v € K. 26.1 

nonnegative inverse eigenvalue problem: Construct a nonnegative matrix with given eigenvalues. 20.3 
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nonnegative integer rank (of nonnegative integer matrix A): The minimum k such that there exists an 
m x k nonnegative integer matrix B andak x n nonnegative integer matrix C with A = BC. 30.3 
nonnegative P-matrix: A P -matrix in which every entry is nonnegative. 35.9 

nonnegative Po-matrix: A Po-matrix in which every entry is nonnegative. 35.9 

nonnegative Po -matrix: A Po )-matrix in which every entry is nonnegative. 35.9 

nonnegative stable (complex square matrix A): The real part of each eigenvalue of A is nonnegative. 9.5 
nonprimary matrix function: Function of a matrix defined using the Jordan Canonical Form using 
different branches for f and its derivatives for two Jordan blocks for the same eigenvalue. 11.1 
nonseparable (graph or digraph): Connected and does not have a cut-vertex. 35.1 

nonsingular (linear transformation): A linear transformation whose kernel is zero. 3.7 

nonsingular (matrix): An invertible matrix. 1.5 

norm (of a vector v, in an inner product space): ./(v, v). 5.1 

norm: See vector norm, matrix norm or specific norm. 

1-norm (of vector x € C”): |[x|]) = [xi] + [x2] +--+ + |xy|. 37.1 

1-norm (of matrix A € C”*"): See maximum column sum norm. 

2-norm (of vector x € C”): ||xl]y = loi |? + [2/2 +--+» + [xnl?. 37-1 

2-norm (of matrix A € C”*"): see maximum column sum norm. 

oo-norm (of vector x € C"): |[Xl]oo = max |x; |. 37.1 


oo-norm (of matrix A): See maximum row sum norm. 

M-norm (where || - || is a vector norm and M is a nonsingular matrix): ||x|| 4 = || Mx||. 37.1 

M-norm (matrix norm): Norm induced by a family of M,-norms for M = {M, : n > 1} a family of 
nonsingular n x n matrices. 37.3 

p-norm (of matrix A € C”*", with p > 1): The matrix norm induced by the (vector) p-norm. 37.1 
p-norm (of vector x € C", with p > 1): |[xllp = (Jail? +--+ + [xn|?)?. 37.1 

normal (complex square matrix or linear operator): Commutes with its Hermitian adjoint. 7.2 

normal equations (for the least squares problem Ax = b): The system A* Ax = A*b. 5.8 

normalized (representation +(m/b?—')b® of a floating point number): b?~! < m < b?. 37.6 
normalized immanant: Function defined by the irreducible character of a partition A of n, specifically, 
flM) = 5 Does, x0(0) Ly miotiy. 31-10 

normalized scaling (of rectangular matrix A): A scaling DAE of A with det(D) = det(E) = 1. 9.6 
NSNS: See nearly sign nonsingular . 

null graph: A graph with no vertices. 28.1 

null space: See kernel. 

nullity: The dimension of the kernel. 2.4, 3.5 

numerical radius (of A € C"*"): w(A) = max{|u| : w € W(A)}. 18.1 

numerical range (of n x n complex matrix A): W(A) = {v* Av|v*v = 1,v € C"}. 7.1, 18.1 

numerical rank (of matrix A, with respect to the threshold t): min{rank(A + E) : ||E ||2 < tT}. 39.9 
numerically orthogonal (vectors x, y): |x7y| < €||x\l2|lyll2- 46.1 

numerically orthogonal matrix: Each pair of its columns are numerically orthogonal. 46.1 

numerically stable (algorithm): Produces results that are roughly as accurate as the errors in the input 
data allow. 37.8 

numerically unstable (algorithm): Allows rounding and truncation errors to produce results that are 
substantially less accurate than the errors in the input data allow. 37.8 


O 


odd (permutation): Can be written as the product of an odd number of transpositions. Preliminaries 
odd (cycle in a sign pattern): The length of the simple or composite cycle is odd. 33.1 

off-diagonal entry: An entry in a matrix that is not a diagonal entry. 1.2 

off-norm (of A): The Frobenius norm of the matrix consisting of all off-diagonal elements of A and a 
zero diagonal. 42.7 
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oh: See big-oh, little-oh. 

one-to-one (linear transformation T): v; 4 v2 implies T(v,) # T(v2). 3.5 

onto (linear transformation): The codomain equals the range. 3.5 

open left half plane: {z € C : re(z) < 0}. Preliminaries 

open right half plane: {z € C : re(z) > 0}. Preliminaries 

open sector (from ray e’” to ray e’*): The set of rays {re’? : r > 0,a@ <0 < PB}. 33.8 

operator norm: See induced norm. 

order (of a graph): The number of vertices in the graph. 28.1 

order (of a square matrix): See size. 

ordered multiplicity list (of real symmetric matrix B): (m,...,m,) where the distinct eigenvalues of B 
are 8} <--- < B, with multiplicities m,...,m,.34.5 

oriented (vertex-edge) incidence matrix (of graph G): A matrix obtained from the incidence matrix of 
G by replacing a 1 in each column by a —1. 28.4 

orthogonal (vectors): Two vectors whose inner product is 0. 5.2 

orthogonal (set of vectors): Any two distinct vectors in the set are orthogonal. 5.2 

J-orthogonal (real matrix B) B TTB = J (where J isa nonsingular real symmetric matrix). 46.5 
orthogonal basis: A basis that is orthogonal as a set. 5.2 

orthogonal complement (of subset S): Subspace of vectors orthogonal to every vector in S. 5.2 
orthogonal iteration: Method for computing the EVD of real symmetric A: given starting n x p matrix 
Xo with orthonormal columns, the sequence of matrices Yx4; = AXz, Year = Xe Regi k = 0,1,2,..., 
where X44) Rx41 is the reduced QR factorization of Y,4). 42.1 

orthogonal matrix: A real or complex matrix Q such that Q? Q = I.5.2,7.1 

orthogonal projection: Projection onto a subspace along its orthogonal complement. 5.4 

orthogonal with respect to symmetric bilinear form f (vectors v,u): f(u,v) = 0. 12.2 

orthonormal (set of vectors): An orthogonal set of unit vectors. 5.2 

orthonormal basis: A basis that is orthonormal as a set. 5.2 

oscillatory (real square matrix A): A is totally nonnegative and A’ is totally positive for some integer 
k> 1.21.1 

outerplanar (graph G): There is an embedding of G into R’ with the vertices on the unit circle and the 
edges contained in the unit disc. 28.2 

overflow: |x| equals or exceeds a threshold at or near the largest floating point number. 37.6 


P 


P-invariant face: Alphabetized under invariant. 

P-matrix: every principal minor is positive. 19.2 

Po-matrix: Every principal minor is nonnegative. 35.8 

P,t-matrix: Every principal minor is nonnegative and at least one principal minor of each order is 
positive. 19.2 

Po,;-matrix: Every principal minor is nonnegative and every diagonal element is positive. 35.8 

pairwise orthogonal (orthogonal projections P and Q): PQ = QP = 0.7.2 

Parter vertex: See Parter—Wiener vertex. 

Parter—Wiener vertex (for eigenvalue A of n x n Hermitian matrix A): j such that w4(j)(A) = aa(A) + 
1. 34.1 

partial completely positive matrix B: Every fully specified principal submatrix of B is a completely pos- 
itive matrix, whenever bj; is specified, then so is bj; and bj; = bj;, and all specified off-diagonal entries 
are nonnegative. 35.4 

partial copositive matrix B: Every fully specified principal submatrix of B is a copositive matrix and 
whenever bj; is specified, then so is bj; and bj; = jj. 35.5 

partial doubly nonnegative matrix B: Every fully specified principal submatrix of B is a doubly nonneg- 
ative matrix matrix, whenever bj; is specified, then so is b;; and b;; = b;;, and all specified off-diagonal 
entries are nonnegative. 35.4 


ij> 


ij 
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partial entry sign symmetric P-matrix B: Every fully specified principal submatrix of B is an entry sign 
symmetric P-matrix and if both b;; and b;; are specified, then b;;bj; > 0 or bj; = bj; = 0. 35.10 
partial entry sign symmetric Po-matrix B: Every fully specified principal submatrix of B is an entry sign 
symmetric Po-matrix and if both b;; and bj; are specified, then bj;bj; > 0 or bj; = bj; = 0. 35.10 
partial entry sign symmetric Po-matrix B: Every fully specified principal submatrix of B is an entry 
sign symmetric Po,,-matrix and if both b;; and bj; are specified, then b;;b;; > 0 or bj; = bj; = 0. 35.10 
partial entry weakly sign symmetric P-matrix B: Every fully specified principal submatrix of B is an 
entry weakly sign symmetric P-matrix and if both b;; and bj; are specified, then b;;b;; = 0. 35.10 
partial entry weakly sign symmetric Po-matrix B: Every fully specified principal submatrix of B is an 
entry weakly sign symmetric Po-matrix and if both b;; and bj; are specified, then bj;b;; = 0.35.10 
partial entry weakly sign symmetric Po,;-matrix B: Every fully specified principal submatrix of B is an 
entry weakly sign symmetric Po,-matrix and if both b;; and bj; are specified, then bj;b;; = 0. 35.10 
partial Euclidean distance matrix B: Every diagonal entry is specified and equal to 0, every fully specified 
principal submatrix of B is a Euclidean distance matrix, and whenever bj; is specified, then so is bj; and 
bji = bij. 35.3 

partial inverse M-matrix B: Every fully specified principal submatrix of B is an inverse M-matrix and 
every specified entry of B is nonnegative. 35.7 

partial M-matrix B: Every fully specified principal submatrix of B is an M-matrix and every specified 
off-diagonal entry of B is nonpositive. 35.6 

partial Mo-matrix B: Every fully specified principal submatrix of B is an My-matrix and every specified 
off-diagonal entry of B is nonpositive. 35.6 

partial matrix: A square array in which some entries are specified and others are not. 35.1 

partial nonnegative P-matrix: Every fully specified principal submatrix is a nonnegative P-matrix and 
all specified entries are nonnegative. 35.9 

partial nonnegative Po-matrix: Every fully specified principal submatrix is a nonnegative Po-matrix and 
all specified entries are nonnegative. 35.9 

partial nonnegative Po,)-matrix: Every fully specified principal submatrix is a nonnegative Po ;-matrix 
and all specified entries are nonnegative. 35.9 

partial P-matrix: Every fully specified principal submatrix is a P-matrix. 35.8 

partial Po-matrix: Every fully specified principal submatrix is a Po-matrix. 35.8 

partial Po |-matrix: Every fully specified principal submatrix is a Po )-matrix. 35.8 

partial positive P -matrix: Every fully specified principal submatrix is a positive P -matrix and all specified 
entries are positive. 35.9 

partial semidefinite ordering: See Loewner ordering. 

partial strictly copositive matrix B: Every fully specified principal submatrix of B is a strictly copositive 
matrix and whenever bj; is specified, then so is bj; and bj; = jj. 35.5 

partitioned (matrix): A matrix partitioned into submatrices by partitions of the row and column 
indices. 10.1 

partly decomposable (n x n (0, 1)-matrix): Has a p x q zero submatrix for positive integers p,q with 
ptq=n.27.2 

path (in a graph): A walk with all vertices distinct. 28.1 

path (in sign pattern A): A formal product of the form y = 4j,;,4j,i, . . . Gizi,,,, where each of the elements 
is nonzero and the index set {i),i2,... , ix41} consists of distinct indices. 33.1 

path-clique (digraph): The induced subdigraph of every alternate path to a single arc is a clique. 35.7 
path-connected (subset S of complex numbers): There exists a path in S (i.e., a continuous function from 
[0,1] to S) from any point in S to any other point in S. Preliminaries 

path cover number (of simple graph G): The minimum number of vertex disjoint induced paths of G 
that cover all vertices of G. 34.2 

pattern (of matrix A): The (0, 1)-matrix obtained from A by replacing each nonzero entry with a 1. 27.1 
pattern block triangular form (partial matrix B): The adjacency matrix of D(B) is in block triangular 
form. 35.1 
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PEIEP: See inverse eigenvalue problem with prescribed entries. 

pencil (defined by A, B € C"*"): A — xB, or pair (A, B). 43.1 

pencil (in D[x]”*”): A matrix A(x) = Ap + xAj, Ao, Ai € D”*". 23.4 

perfect elimination ordering: An elimination ordering that does not produce any fill elements during 
Gaussian elimination. 30.2, 40.5 

perfect matching: A matching M in which each vertex of G is in one edge of M. 30.1 

perfectly-well determined to high relative accuracy (singular values of A): Changing an arbitrary nonzero 
entry axe to Gage, with arbitrary 0 4 0 causes a relative perturbation in 0; bounded by |6| and 1/|0|. 46.3 
period (of access equivalence class J of reducible nonnegative P ): The period of the irreducible matrix 
P{J].9.3 

period (of irreducible nonnegative matrix P ): Greatest common divisor of the lengths of the cycles of the 
digraph of P. 9.2 

period (of reducible nonnegative matrix P): The least common multiple of the periods of its basic 
classes. 9.3 

permanent (of n x n matrix A): per(A) = ess, T]j=1 4ic ci) (also defined for rectangular matrices). 31.1 
permutation: A 1-1 onto function from a set to itself. Preliminaries 

permutation equivalent (matrices A,B): There exist permutation matrices P and Q such that B = 
PAQ.31.1 


permutation invariant (vector norm || - ||): || Px|| = ||x|| for all x and all permutation matrices 
P.37.1 

permutation invariant absolute norm: s function g : R” > Rt that is a norm, and g(x1,... Xn) = 
g(|[xi[,-.-5[Xn|) and g(x) = g(Px) for all x € R” and all permutation matrices P € R"*". 17.3 


permutation matrix: A matrix whose rows are a rearrangement of the rows of the identity matrix. 1.2 
permutation pattern: A square sign pattern matrix with entries 0 and +, where the entry + occurs precisely 
once in each row and in each column. 33.1 

permutation similar (square matrices A,B): There exists a permutation matrix P such that B = 
P-'AP (= P' AP). 27.2 

permutational equivalence (of sign pattern A): A product of the form P, AP, where P, and P are 
permutation patterns. 33.1 

permutational similarity (of sign pattern A) is a product of the form P? AP, where P is a permutation 
pattern. 33.1 

Perron branch (at vertex v ofa tree): The Perron value of the corresponding bottleneck matrix is maximal 
among all branches at v. 36.3 

Perron value (of square nonnegative matrix P ): the spectral radius of P. 9.1 

perturbation (of matrix A): A+ AA 15.1 

perturbation (of scalar 6): 6 + AB 15.1 

perturbation (of vector v): v+ Av 15.1 

Petrie matrix: A (0, 1)-matrix with the 1s in each of its columns occurring in consecutive rows. 30.2 
PIEP: Affine parameterized inverse eigenvalue problem. 20.8 

pinching (of a matrix): Defined recursively. See Section 17.4. 

pivot: An entry of a matrix in a pivot position. 1.3 

pivot column: A column of a matrix that contains a pivot position. 1.3 

pivot position: A position in a matrix in row echelon form that contains a leading entry. 1.3 

pivot row: A row of a matrix that contains a pivot position. 1.3 

planar (graph G): There an embedding of G into R*. 28.2 

PL DU factorization: A factorization of a row permutation of a given matrix as the product of a unit 
lower triangular matrix, diagonal matrix, and a unit upper triangular matrix. 1.6 

PLU factorization: A factorization of a row permutation of a given matrix as the product of a unit lower 
triangular matrix and an upper triangular matrix. 1.6, 38.3 

PO: See potentially orthogonal. 

pointed (cone K): K 1—K = {0}. 8.5, 26.1 
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polar decomposition (of matrix A € C”*” with m > n): A factorization A = UP where P € C"*" is 
positive semidefinite and U € C”%" satisfies U*U = I,. 17.1 

polar form: See polar decomposition. 

polynomial restarting (of the Arnoldi algorithm): Use (A)v instead of vector v when restarting. 44.4 
polytope: See convex polytope. 

positionally symmetric: See combinatorially symmetric. 

positive (linear map ¢ : C"*" + C”*""): 6(A) is positive semidefinite whenever A is positive semidefi- 
nite. 18.7 

positive (real matrix A): All of A’s elements are positive. 9.1 

K -positive (matrix A € R"*"): AK C int K. 26.1 

K -positive (vector v € R"): v € int K. 26.1 

positive definite (matrix): Ann x n Hermitian matrix satisfying x* Ax > 0 for all nonzero x € C”. 5.1, 8.4 
positive definite (real symmetric or Hermitian bilinear form f): f(v,v) > 0 for all nonzero 
veé V. 12.2, 12.5 

positive P-matrix: A P-matrix in which every entry is positive. 35.9 

positive semidefinite (function f : R — C): For eachn € N and all x), x2,...,x, € R, then x nm matrix 
[ f (x; — x;)] is positive semidefinite. 8.5 

positive semidefinite (matrix): Ann x n Hermitian matrix A satisfying x* Ax > 0 for all x € C". 8.4 
positive semidefinite (real symmetric or Hermitian bilinear form f): f(v,v) > 0 for allv € V. 12.2, 12.5 
positive semistable (complex square matrix): See nonnegative stable 

positive set a of edges (in a signed bipartite graph): sgn(@) is +1. 30.1 

positive stable (complex polynomial): Its roots lie in the open right half plane. 19.2 

positive stable (complex square matrix): Its eigenvalues lie in the open right half plane. 9.5, 19.2 
k-potent (square sign pattern or ray pattern A): k is the smallest positive integer such that A = A**!. 33.9 
potentially orthogonal (PO) (sign pattern): Allows an orthogonal matrix. 33.10 

potentially stable (square sign pattern A): Allows stability, ie., some matrix B € Q(A) is stable. 33.4 
Powell-Reid’s complete (row and column) pivoting: A particular pivoting strategy for QR- 
factorization. 46.2 

power method: Method for computing the EVD of A: given starting vector xp, compute the sequences 
Ve = x} Ax;, Xpq1 = Axy/|| Axx ||, k = 0,1,2,... , until convergence. 42.1 

powerful (square sign or ray pattern A): All the powers A!, A’, A®,..., are unambiguously defined. 33.9 
precision: The number of digits, i.e., p in x = + (#7) b°. 37.6 

preconditioner: A matrix M designed to improve the performance of an iterative method for solving the 
linear system Ax = b. 41.1 

preserves: Linear operator T preserves bilinear form f if f(Tu, Tv) = f(u,v) for all u,v, 12.1; Also 
applied to g-sesquilinear form. 12.4 

preserves: Linear operator @ preserves subset of matrices M if 6(M) C M. 22.1 

preserves: Linear operator ¢ preserves relation ~ on matrix subspace V if (A) ~ 6(B) whenever A ~ B, 
A,B Ee V.22.1 

primary decomposition (of a nonconstant monic polynomial q(x) over F): A factorization q(x) = 
(hy(x))™ +++ (h,(x))", where the h;(x),i = 1,...,r are distinct monic irreducible polynomials over F .6.4 


primary factors: The factors in a primary decomposition. 6.4 

primary matrix function: Function of a matrix defined using the Jordan Canonical Form and using the 
same branch for f and its derivatives for each Jordan block for the same eigenvalue. 11.1 

prime (in domain D): A nonzero, nonunit element p such that for any a,b € D, p|ab implies pla or 
p\b. 23.1 

prime (ideal J of a domain): ab € I implies that either a or b is in I. 23.1 

primitive (digraph): There is a positive integer k such that for every pair of vertices u and v, there is a walk 
of length k from u to v. 29.6 

primitive (polynomial p(x) = )7/-5 ajx"—' € Z[x],a9 £0,m > 1): lisa g.c.d. of ag,... 5m. 23.1 
primitive matrix: A square nonnegative matrix whose digraph is primitive. 29.6 
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principal angles (between subspaces X and Y of C’): See Section 17.7 (equivalent to canonical angles, 
Section 15.1.) 

principal ideal: An ideal generated by one element (in a domain). 23.1 

principal ideal domain domain (PID) D: Any ideal of D is principal. 23.1 

principal minor: The determinant of a principal submatrix. 4.2 

principal logarithm (of complex square matrix that has no real eigenvalues < 0): The logarithm with all 
eigenvalues in the strip {z: —m <im(z) <}.114 

principal square root (of a complex square matrix that has no real eigenvalues < 0): The square root with 
each eigenvalue having real part > 0. 11.2 

principal submatrix: A submatrix lying in the same rows as columns. 1.2, 10.1 

principal submatrix at a distinguished eigenvalue 1 (of square nonnegative matrix P ): The principal sub- 
matrix of P corresponding to a set of vertices of [(P ) having no access to a vertex of an access equivalence 
class C that satisfies o(P[C]) > A. 9.3 

principal vectors (between subspaces X and Y of C’): See Section 17.7. 

product (of simple graphs): See (strong) product. 

profile (of sparse matrix A): The number of elements in the envelope of A. 40.5 

projection (of a vector onto V; along V>, assuming V = V; @ V2): The (unique) part of the vector in Vj; 
also, the linear transformation that maps a vector to its projection. 3.6 

proper cone K (in a finite-dimensional real vector space V): A convex cone that is closed, pointed, and 
full. 26.2 

proper subdigraph (ofa digraph T = (V, E)): Asubdigraph I’ = (V’, E’) such that V’ is a proper subset 
of V or E’ is a proper subset of E. 29.1 

properly signed nest: See allows a properly signed nest. 

Property C: An n x n M-matrix A satisfies property C if there exists a representation of A of the form 
A=sI-—  Pwiths > 0, P > Oand f semiconvergent. 9.4 

Property L: A property that complex square matrices A, B have if their eigenvalues a;, 6,, (k = 1,--- ,n) 
may be ordered in such a way that the eigenvalues of xA + yB are given by xa, + yf, for all complex 
numbers x and y. 7.2, 24.3 

PSD square root (of a positive semidefinite matrix): The square root with each eigenvalue > 0 (same as 
principal square root for a positive definite matrix). 8.3 

pseudo-code: An informal computer program used for writing algorithms. 37.7 

€-pseudoeigenvalue (of matrix A € C”*"): Anumber z € C such that there exists a nonzero vector v € C” 
such that || Av — zv|| < é||v||. 16.1 

€-pseudoeigenvector (of matrix A € C”*"): A nonzero vector v € C" such that there exists z € C such 
that || Av — zv|| < é||v||. 16.1 

€-pseudospectral abscissa (ofa complex square matrix): The rightmost extent of the e-pseudospectrum. 16.3 
€-pseudospectral radius (ofa complex square matrix): The maximum magnitude of the e-pseudospectrum. 16.3 
€-pseudospectrum (of complex square matrix A): The set 

{zeC:z€oa(A+E) for some E € C"*" with ||E|| < e}. 16.1 

€-pseudospectrum of a matrix pencil: See Section 16.5. 

€-pseudospectrum of a matrix polynomial: See Section 16.5. 

€-pseudospectrum of a rectangular matrix: See Section 16.5. 


Q 


QMRalgorithm: Iterative method for solving a linear system using non-Hermitian Lanczos algorithm. 41.3 
QR factorization (of matrix A): A= QR where Q is unitary and R is upper triangular. 5.5 


ek R ; 
QR factorization with column pivoting (of A € R”*"): The factorization AI] = Q | i , where IT is a 


permutation matrix, Q is orthogonal, and R is n x n upper triangular. 46.2 
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QR iteration: Method for computing the EVD of A: starting from the matrix Ap = A, the sequence of 
matrices Ay = Qx Rx, Agy1 = Re Qu, kK = 0,1,2,... where Q;, Ry is the QR factorization of Ax. 42.1 
quadrangular (bipartite graph): Simple and each pair of vertices with a common neighbor lie on a cycle 
of length 4. 30.1 

quadratic form (corresponding to symmetric bilinear form f): The map g : V — F defined by 
g(v) = f(v,v), ve V. 12.2 

qualitative class (of complex sign pattern A = A; +7A2): Q(A) = {B) +iB) : B,; € Q(A;) and By € 
Q(A2)}. 33.8 

qualitative class (of real matrix B): The qualitative class of sgn(B). 33.1 

qualitative class (of sign pattern A): The set of all real matrices B with sgn(B) = A. 33.1 

quotient (of vector space V by subspace W): The set of additive cosets of W with operations (v; + W) + 
(v2 + W) = (v;} +¥2) + Wandc(v + W) = (cv) + W force € F.2.3 

quotient field (of an integral domain): The set of equivalence classes of all quotients ¢, b # 0, constructed 
the same way Q is constructed from Z. 23.1 


R 


(R, S)-standard map, (R,c, f)-standard map: Alphabetized under standard. 


radix: The exponentiation base b in floating point number x = + (;7) b°. 37.6 


range (of a linear transformation T : V > W):{T(v): ve V}. 3.4 

range (of a matrix): The span of the columns. 2.4 

rank (of bilinear form f): The rank of a matrix representing f relative to an ordered basis. 12.1. Also 
applied to y-sesquilinear form. 12.4 

rank (of a matrix or linear transformation): The dimension of the range; for a matrix, equals the number 
of leading entries in the reduced row echelon form of the matrix. 1.3, 2.4, 3.5 

rank (of a tensor z): The rank of z is k if z is the sum of k decomposable tensors, but it cannot be written 
as sum of / decomposable tensors for any / less than k. 13.3 

rank revealing (QR factorization (with pivoting) of A): Small singular values of A are revealed by corre- 
spondingly small diagonal entries of R. 46.2 

rank revealing decomposition (of real matrix A): a two-sided orthogonal decomposition of the form 


A=URV' =U ‘ V', where U and V are orthogonal, and R is upper triangular and reveals the 


(numerical) rank of A in the size of its diagonal elements. 39.9 

rational canonical form: See invariant factors rational canonical form or elementary divisors rational 
canonical form. 

rational function: A quotient of polynomials. 23.1 

ray nonsingular (ray pattern A): The Hadamard product X o A is nonsingular for every entry-wise positive 
n X n matrix X. 33.8 

ray pattern: A matrix each of whose entries is either 0 ora ray in the complex plane represented by e’”. 33.8 
ray pattern class (of ray pattern A): Q(A) = {B : bp, = Oiffap, = 0, and otherwise arg bp, = 
arg Apq}. 33.8 


2 
Rayleigh quotient: xs where A is Hermitian and x is a nonzero vector. 8.2 
real generalized eigenspace: See Section 6.3. 
real generalized eigenvector: See Section 6.3. 
real-Jordan block (of size 2k with eigenvalue a + 1): The 2k x 2k matrix having k copies of Mp(@, 8) = 


& ‘| on the (block matrix) diagonal, k — 1 copies of I, = f 


on the first (block matrix) 


: : 0 0 
superdiagonal, and copies of 02 = 0 0 everywhere else. 6.3 
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real-Jordan canonical form (of real square matrix A): A real-Jordan matrix that is similar to A. 6.3 
real-Jordan matrix: A block diagonal real matrix having diagonal blocks that are Jordan blocks or real- 
Jordan blocks. 6.3 

real part (of complex number a + bi): a. Preliminaries 

real structured e-pseudospectrum (of real square matrix A): See Section 16.5. 

real vector space: A vector space over the field of real numbers. 1.1 

reduced digraph (of digraph I"): A digraph whose vertices are the access equivalence classes of I’ (or 
{1,...,k} where k is the number of access equivalence classes) and having an arc from the ith vertex to 
the jth precisely when the ith access class has access to the jth access class. 9.1, 29.5 

reduced QR factorization (of matrix A € C”*", m > n): A= QR where columns of Q are orthonormal 
and R is upper triangular. 5.5 

reduced row echelon form (RREF): A matrix is in RREF if it is in REF, the leading entry in any nonzero 
row is 1, and all other entries in a column containing a leading entry are 0; matrix B is the RREF of matrix 
A if B isin RREF and A and B are row equivalent. 1.3 

reduced singular value decomposition (reduced SVD) (of complex matrix A): A = UxVv*, S= 
diag(o),02,...,0,) € R’*", where 0, > 02 > ... > o, > O and the columns of U e€ C™*" and the 
columns of V € C"*" are both orthonormal. 5.6, 45.1 

BC 


reducible (square matrix A): There is a permutation matrix P such that PAP? = OD 


| , where B, D 


are square. 9.1, 27.3 

reducing eigenvalue (A of A € C”*"): A is unitarily similar to [A] @ Az. 18.2 

REF: See row echelon form. 

(k-)regular (graph): Every vertex has the same degree k. 28.2 

regular (m x n (0,1)-design matrix W): W is balanced and W7'W = t(I + J) for some integer 
t. 32.4 

regular (pencil A — xB): There exists a A € C such that A — 1B is nonsingular. 43.1 

regular (pencil Ay + xA; € D[x]"*"): det( Ap + xA;) is not the zero polynomial, or equivalently, there 
exists a A. € D such that Ap + 2A; is nonsingular. 23.4 

relative backward error of the linear system: See Section 38.1. 

relative condition number: See condition number. 

relative error (in approximation Z to z): ||z — 2||/||z||. 37.4 

requires: If P is a property referring to a real matrix, then a sign pattern A requires P if every real matrix 
in Q(A) has property P. 33.1 

requires unique inertia (sign pattern A): in(B,) = in(B:) for all symmetric matrices B,, By € Q(A). 33.6 
residual vector (of x when solving the linear system Ax = b): The vector r = b — Ax. 38.3 

residual vector (at step k of an iterative method for solving Ax = b): The vector r, = b — Ax;, where xx 
is the approximate solution generated at step k. 41.1 

resolvent (of matrix A € C"*" at a point z ¢ o(A)): The matrix (zI — A)~!. 16.1 

restarted GMRES algorithm, GMRES(j): Restart GMRES every j steps, using the latest iterate as the 
initial guess for the next GMRES cycle. 41.3 

right singular vector: See singular vectors. 

right singular space: See singular spaces. 

ring automorphism of F”*" induced by f: The map ¢ : F"*" > F"*" defined by $([ai;]) = [f (aij)]. 
22.4 

Ritz pair: (6, v) where 0 is the Ritz value for Ritz vector v. 43.3 

Ritz value: The scalar 0 for a Ritz vector. 43.3 

Ritz vector (of matrix of A from subspace S of C”): There isa @ € C such that Av — Ov L S. 43.3 

rook polynomials: Generating functions for the rook numbers defined using permanents. 31.8 
rounding error: In one floating point arithmetic operation, the difference between the exact arithmetic 
operation and the floating point arithmetic operation; in more extensive calculations, refers to the cumu- 
lative effect of the rounding errors in the individual floating point operations. 37.6 
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rounding mode: Maps x € R to a floating point number fl(x); default rounding mode in standard- 
conforming arithmetic is round-to-nearest, ties-to-even. 37.6 

Routh-Hurwitz matrix: See Section 19.2. 

row (of a matrix): The entries of a matrix lying in a horizontal line in the matrix. 1.2 

row echelon form (REF): A matrix is in REF if every zero row is below all nonzero rows and for two 
nonzero rows, the leading entry in the upper row is to the left of the leading entry of the lower row; matrix 
B isa REF of of matrix A if B isin REF and A and B are row equivalent. 1.3 

row equivalent: Matrix A is row equivalent to matrix B if B can be obtained from A by a sequence of 
elementary row operations (equivalently, B = QA for some invertible matrix Q). 1.3 

row equivalent (matrices A, B € D’*"): B = QA for some D-invertible matrix Q. 23.2 

row operation: See elementary row operation. 

row signing (of (real or sign pattern) matrix A): DA where D isa signing. 33.3 

row space: The span of the rows of a matrix. 2.4 

row-stochastic (matrix): A square nonnegative matrix having all row sums equal to 1. 9.4 

row sum vector (of a matrix): The vector of row sums. 27.4 

RRD (of real matrix A): A decomposition A = X DY? with D a diagonal matrix and X and Y full column 
rank, well-conditioned matrices. 46.3 

RREF: See reduced row echelon form. 


S 


S-matrix: An n x (n + 1) matrix B such that it is an S*-matrix and the kernel of every matrix in Q(B) 
contains a vector all of whose coordinates are positive. 33.3 

S*-matrix: An n x (n+ 1) matrix B such that each of the n + 1 matrices obtained by deleting a column 
of B is an SNS matrix. 33.3 

SAP: See spectrally arbitrary pattern. 

scalar: An element of a field. 1.1 

scalar matrix (transformation): A scalar multiple of the identity matrix (transformation). 1.2, 3.2 
scaling (of a real matrix A): A matrix of the form D, AD) where D, and D) are diagonal matrices with 
positive diagonal entries. 9.6, 27.6 

Schatten-p norm (of A € C”*"): ||Alls,p = (0, of (A))"””. 17.3 

Schur complement: A matrix defined from a partitioned matrix. 4.2, 10.3 

Schur product: See Hadamard product. 

score vector: The row sum vector of a tournament matrix. 27.5 

SDR: See system of distinct representatives. 

Seidel matrix (of simple graph G): The matrix J — I — 2Ag. 28.4 

Seidel switching: An operation on simple graphs. 28.4 

self-adjoint: See Hermitian. 

self-inverse sign pattern: An S*NS-pattern A such that B~! € Q(A) for every matrix B € Q(A). 33.2 
semiconvergent (square nonnegative matrix P): lim,,.>. P™” exists. 9.3 

semipositive (real matrix A): A is nonnegative and some element is positive. 9.1 

K-semipositive (matrix A € R’*"): A is K-nonnegative and A # 0. 26.1 

K-semipositive (vector v € R"):v € K andv¥ 0. 26.1 

semisimple: An eigenvalue having algebraic multiplicity equal to its geometric multiplicity. 4.3 
semistable matrix: Either positive semistable (i.e., nonnegative stable) or negative semistable, depending 
on section. 

separation (between two square matrices A; and A2): inf) x),=1 || X Ai — A2X|l2. 15.1 

g-sesquilinear form (on vector space V over field F with automorphism g): A map f from V x V 
into F that satisfies f(au; + bu2,v) = af(uj,v) + bf(uz,v) and f(u,av, + bv.) = y(a) f(wv) + 
y(b) f (u, v2). 12.4 

sesquilinear form (on a complex vector space): A y-sesquilinear form where g is complex conjugation. 12 
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set of Hermitian matrices associated with graph G: H(G) = {B € H,| G(B) = G}. 34.1 

set of symmetric matrices associated with graph G: S(G) = {B € S,| G(B) = G}. 34.1 

sgn: See sign. 

Shannon capacity: A parameter of a simple graph. 28.5 

shift: The scalar yz in a shifted matrix A — wl. 42.1 

shifted matrix (of matrix A): The matrix A — yl, where p is the shift. 42.1 

shifted QR iteration: QR iteration of the shifted matrix. 42.1 

sign (denoted sign, of acomplex number): A nonzero complex number; sign(0) = 1, otherwise the complex 
number with the same argument having absolute value 1. Preliminaries 

sign (denoted sign, of a complex square matrix): A function defined from the Jordan Canonical Form of 
the matrix. See Section 11.6. 

sign (denoted sgn, of a permutation): 1, if the permutation is even and —1, if the permutation is odd. 
Preliminaries 

sign (denoted sgn, of a real number): +, 0, — according as the number is > 0, = 0, < 0. Preliminaries 
sign (denoted sgn, of a set a of edges in a signed bipartite graph): The product of the weights of the edges 
in a. 30.1 

sign (of a simple cycle in a sign pattern A): The product of the entries in the cycle. 33.1 

sign-central: A sign pattern matrix that requires centrality. 33.11 

sign nonsingular (SNS) (square sign pattern A): Every matrix B € Q(A) is nonsingular. 33.2 

sign pattern: A matrix whose entries are in {+, 0, —}. 33.1 

sign pattern class: See qualitative class. 

sign pattern matrix: See sign pattern. 

sign pattern of real matrix B: The sign pattern whose entries are the signs of the corresponding entries 
in B. 33.1 

sign potentially orthogonal SPO (square sign pattern): Does not have a zero row or zero column and 
every pair of rows and every pair of columns allows orthogonality. 33.10 

sign semistable (square sign pattern A): Every matrix B € Q(A) is semistable 33.4 

sign singular (square sign pattern A): Every matrix B € Q(A) is singular. 33.2 

sign-solvable (system of linear equations Ax = b): For each A € Q(A)and foreach b € Q(b), the system 
Ax = b is consistent and {x : there exist A € Q(A) and b € Q(b) with Ax = b} is entirely contained in 
one qualitative class. 33.3 

sign stable (square sign pattern A): Requires stability, ie., every matrix B € Q(A) is stable. 33.4 

sign symmetric (square matrix A): det A[a@, 6] det A[B,a] > 0, Va, B C {1,...,n}, |a| = |B]. 19.2 
signature (of a real symmetric or Hermitian bilinear form): The signature of a matrix representing the 
form relative to some basis. 12.2, 12.5 

signature (of real symmetric or Hermitian matrix A): The number of positive eigenvalues minus the 
number of negative eigenvalues. 12.2, 12.5 

signature (of a composite cycle y in sign pattern A): (—)©G—-D where i; are the lengths of the simple 
cycles in y. 33.1 

signature matrix: A +1 diagonal matrix. 32.2 

signature pattern: A diagonal sign pattern matrix, each of whose diagonal entries is + or —. 33.1 
signature similarity (of the square sign pattern A): A product of the form SAS, where S is a signature 
pattern. 33.1 

signed bigraph (of real matrix A): The weighted graph obtained from the bigraph of A weighting the edge 
{i, j’} by +1 ifaj; > 0, and by —1 ifa;; < 0. 30.2 

signed bipartite graph: A weighted bipartite graph with weights X = {—1, 1}. 30.1 

signed 4-cockade: A 4-cockade whose edges are weighted by +1 in such a way that every 4-cycle is 
negative. 30.2 

signed digraph (of an n x n sign pattern A): The digraph with vertex set {1,2,...,} where (i, j) is an 
arc (bearing aj; as its sign) iffa;; A 0. 33.1 


significand: The (base b) integer m in floating point number x = + ( pra ) b°. 37.6 
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signing (of order k): A nonzero (0, 1, —1)- or (0, +, —)-diagonal matrix of order k. 33.3 

signless Laplacian matrix (of graph G): The matrix D + Ag, where D is the diagonal matrix of vertex 
degrees and Ag is the adjacency matrix of G. 28.4 

similar (to matrix A): Any matrix of the form C7! AC. 2.4 

similarity scaling (of square matrix A): A scaling DAD™! of A. 9.6 

simple (cycle) (in a digraph): The induced subdigraph of the vertices in the cycle is the cycle. 35.1 
simple cycle (in sign pattern A): See k-cycle (in sign pattern A). 

simple (digraph): A digraph with no loops (a digraph does not have multiple arcs). 29.1 

simple (eigenvalue): An eigenvalue having algebraic multiplicity 1. 4.3 

simple (graph): A graph with no loops where each edge has multiplicity at most one. 28.1 

simple (walk) (in a digraph): A walk in which all vertices, except possibly the first and last, are distinct. 29.1 
simple row operation: A generalization of an elementary row operation used in domains. 23.2 

sine (of a complex square matrix A): The matrix defined by the sine power series, 

sin(A) = A A Peed ce AX 4... 13.5 

single precision: Typically, floating point numbers have machine epsilon roughly 107’ and precision 
roughly 7 decimal digits. 37.6 

singular (matrix or linear operator): Not nonsingular. 1.5, 3.7 

singular (pencil): A pencil that is not regular. 23.4, 43.1 

singular space (right or left) (of complex matrix A): The subspace spanned by the right (or by the left) 
singular vectors of A. 45.1 

singular-triplet (of A ¢ C”*”): A (left singular vector, singular value, right singular vector) triplet. 15.2 
singular value decomposition (SVD) (of matrix A € C”*"): A= UZV*, 

xX = diag(o),02,...,0)) € R”*", p = min{m,n}, where 0, > 02. => ... => Op = 0 and both 
U = [uy,uy,...,Un] € C”*” and V = [v1,v2,...,vn] € C"*" are unitary. Also forms with other 
dimensions. 5.6, 45.1, 17.1 

singular value vector (of complex matrix A): The vector of singular values of A in nonincreasing order. 17.1 
singular values (of a complex matrix): The diagonal entries 0; of & in a singular value decomposition. 
5.6, 45.1, 17.1 

singular vectors (of a complex matrix): Left: the columns of U, and right: the columns of V, both in a 
singular value decomposition. 5.6, 45.1 

size (of matrix A): m x n, where A isan m x n matrix; also m ifn = m. 1.2 


Skeel condition number of the linear system Ax = b: cond(A, x) = ATL LAL illo 38.1 


Skeel matrix condition number (of matrix A): cond(A) = || |A7'| | Al low 38.1 

skew-Hermitian (matrix): A real or complex matrix equal to the negative of its conjugate transpose. 1.2, 7.2 
skew-symmetric (matrix): A matrix equal to the negative of its transpose. 1.2 

Smith invariant factors (of A € F"*"): The nonconstant polynomials on the diagonal of the Smith 
normal form of xI — A. 6.5 

Smith normal form (of M é€ F [x]"*"): The Smith normal matrix obtained from M by elementary row 
and column operations. 6.5 

Smith normal form (over GCD domain D, ): Matrix B € be is in Smith normal form if 

B =diag(by,...,b,,0,...,0), bj #0 fori =1,...,r and bj_,|b; fori = 2,... ,r. 23.2 

Smith normal matrix (in F[x]"*"): A diagonal matrix diag(1,...,1,a)(x),...,as;(x),0,...,0), where 
the a;(x) are monic nonconstant polynomials such that a; (x) divides aj4;(x) fori = 1,...,s — 1.6.5 
SNS: See sign nonsingular. 

SNS: See strong sign nonsingular. 

solution: An ordered tuple of scalars that when assigned to the variables satisfies a system of linear 
equations. 1.4 

solution set: The set of all solutions to a system of linear equations. 1.4 

span, Span (noun): The set of all linear combinations of the vectors in a set. 2.1 

span, spans (verb): A set S C V spans V if V = Span(S). 2.1 

spanning subgraph (of a connected graph (V, E)): A connected subgraph (V’, E’) with V’ = V. 28.1 
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spanning tree (of a connected graph): A spanning subgraph that is a tree. 28.1 

sparse (matrix A): A matrix for which substantial savings in either operations or storage can be achieved 
when the zero elements of A are exploited during the application of Gaussian elimination to A. 40.2 
sparse approximate inverse (of matrix A): A sparse matrix M7! constructed to approximate A7!. 41.1 
sparsity pattern: See sparsity structure. 

sparsity structure (of matrix A): The set of indices of nonzero elements of A. 40.3 

special linear group (of order n over F ): The subgroup of the general linear group consisting of all matrices 
that have determinant 1. 67.1 

spectral absolute value (of complex matrix A): (A*.A)!/?. 17.1 

spectral norm (of complex matrix A): || All, = /p(A*A) = max{||Avll2 : |lvll2 = 1} = the largest 
singular value of A. 7.1, 37.3 

spectral radius (of A € C”*"): max{|A| : A an eigenvalue of A}. 4.3 

spectrally arbitrary pattern (SAP): A sign pattern A of size n such that every monic real polynomial of 
degree n can be achieved as the characteristic polynomial of some matrix B € Q(A). 33.6 

spectrum (of a matrix): The multiset of all eigenvalues of a matrix, each eigenvalue appearing with its 
algebraic multiplicity. 4.3 

spectrum (ofa graph): The spectrum of its adjacency matrix. 28.3 

speed (of a processor): The number of floating-point operations per second (flops). 42.9 

splitting: See preconditioner. 

SPO: See sign potentially orthogonal. 

square matrix: The number of columns is the same as the number of rows. 1.2 

square root (of a square complex matrix A): A matrix B such that B? = A. 11.2 (See also principal square 
root, PSD square root) 

stable (algorithm): See numerically stable. 

stable matrix: Either positive stable or negative stable, depending on section; positive in Section 9.5 and 
Section 19.2. 

standard basis (for F” or F”*"): The set of vectors or matrices having one entry equal to 1 and all other 
entries equal to 0. 2.1 

standard-conforming (floating point system): conforming to IEEE standard. 37.6 

standard inner product (in R" (or C”)): (u,v) = u’v ((u,v) = v*u). 5.1 

standard linear preserver problem: A linear preserver problem where the preservers take one of the 
standard forms. 22.2 

(R,c, f)-standard map: For A € F"*", @(A) = cRAR7! + f(A)I, or $(A) =cRA'R! + f(A)I.R 
invertible, c nonzero, f a linear functional. 22.2 

(R, S)-standard map: For A € F”*", (A) = RAS, orm = n and $(A) = RATS. R and S must be 
invertible and may be required satisfy some additional assumptions. 22.2 

standard matrix of transformation T : F" — F”™: The matrix of T with respect to the standard bases 
for F", F™. 3.3 

star on n vertices: a tree in which there is a vertex of degree n — 1. 34.6 


state (of n x n stochastic matrix P): An indexi € {1,...,n}. 9.4 

stationary distribution (of stochastic matrix P): Nonnegative vector 2 that satisfies 771 = 1 and 
T T 

me P=n’.94 


k-step Arnoldi factorization: See Arnoldi factorization. 

stochastic (square nonnegative matrix P): Same as row-stochastic; the sum of the entries in each row 
is 1.9.4 

stopping (matrix): A transient substochastic matrix. 9.4 

strict column signing (of (real or sign pattern) matrix A): AD where D isa strict signing. 33.3 

strict row signing (of (real or sign pattern) matrix A): DA where D is a signing. 33.3 

strict signing: A signing where all the diagonal entries are nonzero. 33.3 

strictly block lower triangular (matrix A): A’ is strictly block upper triangular. 10.3 
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strictly block upper triangular (matrix): A block upper triangular matrix in which all the diagonal blocks 
are 0. 10.3 

strictly copositive (matrix A): x’ Ax > 0 for all x > 0 and x ¥ 0. 35.5 

strictly diagonally dominant (n x n complex matrix A): |aji|_ > Pigs la;;| fori = 1,..., 
n.95 

strictly equivalent (pencils A — xB and C — xD): If there exist nonsingular matrices S; and S) such that 
C-—AD = S\(A —AB)S) for allA € C. 43.1 

strictly equivalent (pencils A(x), C(x) € D[x]*"): C(x) = QA(x)P for some D-invertible matrices 
P,Q. 23.4 

strictly lower triangular (matrix A): A’ is strictly upper triangular 10.2 

strictly upper triangular (matrix): A matrix such that every entry having row number greater than or 
equal to column number is zero. 10.2 

strictly unitarily equivalent (pencils A — x B and C — xD): strictly equivalent pencils in which S,, S) can 
be taken unitary. 43.1 

Strong Arnold Hypothesis: Satisfied by a real symmetric matrix M provided there does not exist a real 
symmetric nonzero matrix X such that MX = 0, Mo X = 0,] 0 X = 0. 28.5 

strong combinatorial invariant (of a matrix): A quantity or property that does not change when the rows 
and/or columns are permuted. 27.1 

strong Parter vertex: a Parter—-Wiener vertex j for an eigenvalue A of an A € 7{(G) such that A occurs as 
an eigenvalue of at least three direct summands of A(j). 34.1 

(strong) product (of simple graphs G = (V, E), G’ = (V’, E’)): The simple graph with vertex set V x V’, 
where two distinct vertices are adjacent whenever in both coordinate places the vertices are adjacent or 
equal in the corresponding graph. 28.1 

strong sign nonsingular (S*NS) (square sign pattern A): A is an SNS-pattern such that the matrix B~! is 
in the same sign pattern class for all B € Q(A). 33.2 

strongly connected: A digraph whose vertices all lie in a single access equivalence class. 9.1, 29.5 
strongly connected components (of a digraph): The subdigraphs induced by the access equivalence 
classes. 29.5 

strongly inertia preserving (real square matrix A): The inertia of AD is equal to the inertia of D for every 
real diagonal matrix D. 19.5 

strongly nonsingular (square matrix): All its principal submatrices are nonsingular. 47.6 

¢ strongly preserves (subset of matrices M): 6(M) = M. 22.1 

¢ strongly preserves (relation ~ on matrix subspace V): For every pair A, B € V, we have (A) ~ $(B) 
ifand only if A ~ B. 22.1 

strongly regular: A simple graph with parameters (,k,A,) that has n vertices, is k-regular with 
1 < k < n — 2, every two adjacent vertices have exactly A common neighbors, and every two distinct 
nonadjacent vertices have exactly 4. common neighbors. 28.2 

strongly stable: See additive D-stable. 

subdigraph (of a digraph T = (V, E)): A digraph I’ = (V’, E’) with V’ C V and E’ C E. 29.1 
subgraph (of a graph (V, E)): A graph G’ = (V’, E’) with V’ C Vand E’ C E.. 28.1 

submatrix: A matrix lying in certain rows and columns of a given matrix. 1.2, 10.1 

D-submodule: A nonempty set N C M that is closed under addition and multiplication by (D) scalars. 23.3 
submultiplicative (norm || - || on C”*”): A vector norm satisfying || AB|| < || A|l|| Bl] for all A, B e C"*" 
(satisfies the conditions of a matrix norm except not required to be part of a family). 18.4 

subordinate (matrix norm): See induced norm. 

(k-th)-subpermanent sum (of matrix A): The sum of the permanents of all order k submatrices 
of A. 31.7 

subpattern (of a sign pattern): A sign pattern obtained by replacing some (possibly none) of the nonzero 
entries with 0. 33.1 

subspace: Subset of a vector space V that is itself a vector space under the operations of V. 1.1 
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subspace iteration: See orthogonal iteration. 
substochastic (square nonnegative matrix P ): The sum of the entries in each row is < 1.9.4 
subtractive cancellation of significant digits: Occurs in floating point sums when the relative error 
in the rounding-error-corrupted approximate sum is substantially greater than the relative error in the 
summands. 37.6 
sum (of subspaces): W, +---+ Wy = {wi +--+ we iw; € Wi}. 2.3 
sum norm: See 1-norm. 
sup-norm: See oo-norm. 
support (of rectangular matrix A or vector): The set of indices ij with a;; 4 0.9.6 
support line (of convex set S): A line é that intersects 0S such that S lies entirely within one of the closed 
half-planes determined by €. 18.2 
G supports rank decompositions: Each matrix A € M(G) is the sum of rank(A) matrices in M(G) each 
having rank 1 (also a variant for signed bipartite graphs). 30.3 
surjective: See onto. 
SVD: See singular value decomposition. 
switching equivalent: A simple graph that can be obtained from another simple graph by a Seidel switch- 
ing. 28.4 
Sym: The map Sym(v, ®@--- @ vz) = mt Domes, Vr(1) @ +++ @ Vik). 13.6 
sym multiplication: (v; V --- V Vp) V (Vp41 V +++ V Vp4q) = Vi V+ ++ V Vp4q- 13.7 
symbol (of a family of Toeplitz matrices): The function a(z) = 5> a,z* where the a, are the constants of 
the Toeplitz matrices. 16.2 
symbol curve (of a Toeplitz family): The image of the complex unit circle under the symbol. 16.2 
symmetric (bilinear form f): f(u,v) = f(v,u) forallu,v € V. 12.2 
symmetric (digraph = (V, E)): (i, 7) € E implies (j,i) € E for alli, j € V. 35.1 
symmetric (form): A multilinear form that is a symmetric map. 13.4 
symmetric (map w € L"(V;U)): W(va(1)5--- > Va(m)) = WV... 5 Vn), for all permutations 7. 13.4 
symmetric (matrix): A matrix equal to its transpose. 1.2 
symmetric algebra (on vector space V): V V= QD (vi v) with sym multiplication. 13.9 

keN 
symmetric inertia set (of a symmetric sign pattern A): in(A) = {in(B): B = B? € Q(A) }; 33.6 
symmetric matrices associated with graph G: See set of symmetric matrices associated with graph G. 
symmetric maximal rank (of symmetric sign pattern A): max{rankB : BT = Band B € Q(A)}. 33.6 
symmetric minimal rank (of symmetric sign pattern A): min{rankB : B’ = Band B € Q(A)}. 33.6 
symmetric power (of a vector space): See symmetric space. 
symmetric product (of vectors Vv), ...5 Vin): Vi V-++ V Vm = m!Sym(v) ® --- @ vz). 13.6 
symmetric rank revealing decomposition (SRRD) (of symmetric real matrix H): A decomposition 
H = XDX', where D is diagonal and X is a full column rank well-conditioned matrix. 46.5 
symmetric space (of vector space V): \/" V =Sym(®)” V). 13.6 
symmetric scaling (of a matrix): A scaling where D; = D>. 9.6, 27.6 
system of distinct representatives (SDR): For the finite sets S,, S2,..., S,, a choice of x), x2,...,X, with 
the properties that x; € S; for eachi and x; 4 x; whenever i 4 j. 31.3 
system of linear equations: A set of one or more linear equations in the same variables. 1.4 


T 


tensor: An element of a tensor product. 13.2 


tensor algebra (on vector space V): ®&) VS ‘<) (@' v). 13.9 
keN 
tensor multiplication: (v; © --- @ Vp) @ (Vp41 @ +++ @ Vp4q) = Vi @ +++ @ Vp4q- 13.7 
tensor power: A tensor product of copies of one vector space. 13.2 
tensor product (of matrices): See Kronecker product. 
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tensor product (of vector spaces Vj,..., Vin): a multilinear map satisfying a universal factorization 
property, See Section 13.2. 

term rank (ofa (0, 1)-matrix A): The largest size ofa collection of 1s of A with no two 1s in the sameline. 27.1 
term rank (of sign pattern A): The maximum number of nonzero entries of A no two of which are in the 
same row or same column. 33.6 

tight sign-central: A sign-central pattern A for which the Hadamard product of any two columns contains 
a negative element. 33.11 

TN: See totally nonnegative. 

Toeplitz (matrix): A matrix whose entries are constant along each sub- and super- diagonal, i.e., the value 
of the i, j-entry is the constant a; whenever i — j = k. 16.2, 48.1 

Toeplitz-block matrix: A block matrix A = [Aj;;] where each block Aj; is ann x n Toeplitz matrix. 48.1 
Toeplitz inverse eigenvalue problem (ToIEP): Given A),...,4, € R, finde = [cy,..., Cn]! € R" such 
that [ t;; | es with f;; = ¢);_ 4; has spectrum {A1,... An}. 20.9 

ToIEP: See Toeplitz inverse eigenvalue problem. 

total least squares problem: See Section 39.1. 

total signed compound (TSC) (sign patterns): Every square submatrix of every matrix A € Q(S) is either 
sign nonsingular or sign singular. 46.3 

total support: An n x n (0, 1)-matrix A has total support if A 4 0 and each 1 of A is ona diagonal of A. 27.2 
totally nonnegative (real matrix A): Every minor is nonnegative. 21.1 

totally positive (real matrix A): Every minor is positive. 21.1 

totally unimodular (real matrix A): All minors of A are from {—1, 0, 1}). 46.3 

tournament: Digraph of a tournament matrix. 27.5 

tournament matrix: A (0,1)-matrix with 0 diagonal and exactly one of a;;,a;; equal to 1 for alli 4 j.27.5 
TP: See totally positive. 

trace: The sum of all the diagonal entries of the matrix. 1.2 

trace-minimal (graph G ina family F of graphs): The trace-sequence of the adjacency matrix of G is least 
in lexicographic order among all graphs in F. 32.7 

trace norm (of A € C’*”): The sum of the singular values of A. 17.3 

trace-sequence (of n x n matrix A): (tr(A), tr(A’),--- , tr(A”)). 32.7 

transient class (of a stochastic matrix P ): an access equivalence class of P that is not ergodic. 9.4 
transient matrix: See convergent. 

transient state: A state in a transient class. 9.4 

transition matrix: See change-of-basis matrix. 

transitive tournament matrix A: 4;; = aj, = 1 implies aj, = 1.27.5 

transpose (of a linear transformation): A specific linear transformation from the dual of the codomain to 
the dual of the domain. 3.8 

transpose (of m x n matrix A): The n x m matrix B = [b;;] where bjj = aj;. 1.2 

transposition: A 2-cycle. Preliminaries 

tree (digraph): A digraph whose associated graph is a tree. 29.1 

tree (graph): A connected graph with no cycles. 28.1 

tree sign pattern (t.s.p.): A combinatorially symmetric sign pattern matrix whose graph (suppressing 
loops) is a tree. 33.5 

triangular (matrix): Upper or lower triangular. 10.2 

triangular property (of class of matrices X): Whenever A is a block triangular matrix and every diagonal 
block is an X-matrix, then A is an X matrix. 35.1 

triangular system: A linear system Tx = b where T is a triangular matrix. 38.2 

triangular totally positive (ATP): A triangular matrix all of whose nontrivial minors are positive. 21.2 
tridiagonal matrix: A square matrix A such that aj; = 0 if |i — j| > 1. 

trivial face (F of cone K): F = {0} or F = K. 26.1 

trivial linear combination: A linear combination in which all the scalar coefficients are 0 (or over the 
empty set). 2.1 
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truncation error: The error made by replacing an infinite process by a finite process. 37.6 
TSC: See total signed compound. 

t.s.p. See tree sign pattern. 

two-sided Lanczos algorithm: See Section 41.3. 

type I (tree): Has exactly one characteristic vertex. 36.3 

type II (tree): Has two characteristic vertices. 36.3 


U 


UED: See unique factorization domain. 

u.i.: See unitarily invariant. 

underflow: fl(x) = 0 for x £ 0. 37.6 

unicyclic: A graph containing precisely one cycle. 36.2 

unimodular (matrix over a domain): See D-invertible. 

union (of two graphs G = (V,E) and G’ = (V’,E’)): The graph with vertex set V U V’, and edge 
(multi)set E U E’. 28.1 

unique factorization domain (UFD): Any nonzero, nonunit element a can be factored as a product of 
irreducible elements a = p, --- p;, and this factorization is unique within order and unit factors. 23.1 
unisigned (real or sign pattern vector): Not balanced. 33.3 

unit (in a domain): An element a such that a divides 1. 23.1 

unit lower triangular matrix: A lower triangular matrix such that all diagonal entries are equal to one. 
unit round: u = inf {6d > 0| fl(1 +5) > 1}. 37.6 

unit upper triangular matrix: An upper triangular matrix such that all diagonal entries are equal to one. 1.2 
unit vector: A vector of length 1.5.1 

unital (linear map ¢ : C"*" > C™*™): 6(I,) = In. 18.7 

unitarily equivalent: See unitarily similar. 

unitarily invariant (vector norm ||- || on C”*”): || Al| = || U AV]| for any unitary U € C”™*” and V € C"*" 
and any A € C”*". 17.3 

unitarily similar (matrices A and B): There exists a unitary matrix U such that B = U* AU. 7.1 
unitarily similarity invariant (vector norm ||- || on C”*”): || U* AU || = || Al] forall A € C”*” and unitary 
U,V € C"*". 18.4 

unitary matrix: A matrix U such that U*U = I.5.2,7.1 

unknown vector: The vector of variables of a system of linear equations. 1.4 

unreduced upper Hessenberg matrix: An upper Hessenberg matrix A such that aj41,; 4 0 for 
j=l,...,n—1.43.2 

unstable (algorithm): See numerically unstable. 

updating (QR factorization): See Section 39.7. 

upper Hessenberg (matrix A): a;; = 0, for alli > j + 2,1 <i, j <n. 10.2, 43.2 

upper triangular (pencil A — xB): Both A and B are upper triangular. 43.1 

upper triangular matrix: A matrix such that every entry having row number greater than column number 
is zero. 1.2, 10.2 


V 


(v,k,A)-design: See 2-design. 

valency: See degree. 

Vandermonde matrix: A matrix having each row equal to successive powers of a number. 48.1 

variables (of a linear equation): The unknowns. 1.4 

vec-function (of matrix A): The vector formed by stacking the columns of A on top of each other in their 
natural order. 10.4 

vector: An element of a vector space. 1.1 
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vector norm: A real-valued function || - || on R” or C” such that for all vectors x, y and all scalars a: (1) 
||x|| = 0, and ||x|| = 0 implies 0; (2) |]orx|] = Jo|||xl]; (3) [Ix + yll < IIxll + llyll. 37-1 
vector seminorm: A real-valued function || - || on R” or C” such that for all vectors x, y and all scalars a: 


(1) [xl] = 05 (2) lor] = Joel xls (3) Ix + yll < [xl + Ilyll. 37.2 

vector space (over field F): A nonempty set V with two operations, addition and scalar multiplication, 
such that V is an abelian group under addition, scalar multiplication distributes over addition, scalar 
multiplication is associative, and the multiplicative identity of F acts as the identity on V. 1.1 

vertex: An element of the vertex set of a graph or digraph 28.1, 29.1 

vertex coloring: A partition of the vertex set of a graph into cocliques. 28.5 

vertex-edge incidence matrix: See incidence matrix. 30.1 

vertex independence number: The largest order of a coclique in G. 28.5 

vertices: Plural of vertex. 

Volterra—Lyapunov stable: See Lyapunov diagonally stable. 


W 


walk (in a digraph): A sequence of arcs (vo, V1), (V1,V2),.-.5(Ve—1, Vx) (the vertices need not be 
distinct). 29.1 

walk (in a graph): An alternating sequence (Vj, €;,5 Vi,» i,» ++» Ci,s Vi, ) of vertices and edges, not necessarily 
distinct, such that Vij and v;, are endpoints of ei, for j =1,...,€.28.1 

walk product: we Ay, v; where A is a square matrix and (Vo, v1), (V1, ¥2),.- +» (Vk—15 Vk) is a walk in the 


digraph of A. 29.3 

walk-regular (graph): For every vertex v the number of walks from v to v of length 2, depends only on @ 
(not on v). 28.2 

weak combinatorial invariant (of a matrix): A quantity or property that does not change when the rows 
and columns are simultaneously permuted (by the same permutation). 27.1 

weakly majorizes: Real sequence a weakly majorizes f if for all k, the sum of the first k entries of a in 
decreasing order is > the sum of the first k entries of 6 in decreasing order. Preliminaries 

weakly numerically stable (algorithm): The magnitude of the forward error is roughly the same as the 
magnitude of the error induced by perturbing the data by small multiple of the unit round. 37.8 
weakly sign symmetric (square matrix A): det A[a, 8] det A[B,a] > 0, Va, B C {1,...,n}, lo] = |B] = 
la B| + 1.19.2 

weakly unitarily invariant: See unitarily similarity invariant. 

weighted bigraph (of matrix A): The bigraph of A with the weight of {i, j’} = a;;. 30.2 

weighted bipartite graph: A simple bipartite graph with a weight function on the edges. 30.1 

weighted digraph: A digraph with a weight function on the arcs. 29.1 

weight function: A function from the edges or arcs of a graph. bipartite graph, or digraph to Rt 36.4, 
30.1, 29.1 

weighted graph: A simple graph with a weight function. 36.4 

weighted least squares problem: See Section 39.1. 

Wiener vertex: See Parter—Wiener vertex. 

well-conditioned (input z for P): Small relative perturbations of z cause small relative perturbations of 
P(z). 37.4 

Wilkinson’s shift: The shift is the eigenvalue of the bottom right 2 x 2 submatrix of T which is closer 
tO tn. 42.3 


Z 


Z-matrix (square real matrix): All off-diagonal elements are nonpositive. 9.5, 19.2 
zero completion (of a partial matrix): Obtained by setting all unspecified (off-diagonal) entries to 0 
(partial matrix must have all diagonal entries specified). 35.6 
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zero divisor (in a ring R): A nonzero element a € R such that there exists a nonzero b € R withab = 0 
or ba = 0. Preliminaries 

zero-free diagonal (property of matrix A): All the diagonal elements of A are nonzero. 40.3 

zero line (in a matrix): A line of all zeros. 27.1 

zero matrix: A matrix with all entries equal to zero. 1.2 

(zero) pattern (of a matrix): See pattern. 

zero pattern (of sign pattern A): The (0, +)-pattern obtained by replacing each — entry in A bya +. 33.2 
zero transformation: A linear transformation that maps every vector to zero. 3.1 


Notation Index 


This notation index covers most of the terms defined in Chapters 1 to 49. It does not cover some terminol- 
ogy used in a single section (including most of the terminology that is specific to a particular application 
(Chapters 50 to 70)), nor does it cover most of the terminology used in computer software (Chapters 71 
to 77). 

Notation is in “alphabetical” order. If you are looking for something done to a matrix, like the transpose, 
look under A. Ifyouare looking for something done toa field, look under F . Ifyouare looking for something 
done to a linear transformation, look under T. If you are looking for something done to a vector or vector 
space, look under V. 

The meaning of a symbol depends on what it is applied to; e.g., o(A), where A is a matrix, is the spectral 
radius of A, whereas p(G), where G is a group, is a representation of the group. 

Warning: Tilde and hat, as in A, Q, frequently have the meanings perturbation and reduced, respectively, 
but are also redefined in some chapters to mean other things. 

For most symbols, the section where the symbol is introduced is listed at the end of the definition. 


Onn the zero matrix (in F’*”), can be shortened to 0. 1.2 

1B all 1s vector in F” can be shortened to 1. 

A= [aij] matrix A and its elements. 1.2 

Al transpose of matrix A. 1.2 

A complex conjugate of matrix A. 1.2 

A* Hermitian adjoint (conjugate-transpose) of matrix A. 1.2 

A! inverse of square matrix A. 1.5 

At Moore-—Penrose pseudo-inverse of matrix A. 5.7 

At group inverse of square matrix A. 9.1 

Ala, B] submatrix of A with row indices in aw and column indices in f. 1.2, 10.1 

Ala] = A[a, a], principal submatrix, also denoted A[1, 2, 3] for A[{1, 2, 3}]. 1.2, 10.1 

A(qa, B) submatrix of A with row indices not in a and column indices not in f. 1.2, 
10.1 

A(a) = A(a@,a@), principal submatrix with row and column indices not in q@. 1.2, 
10.1 

Aik, j:l submatrix A[{i,i+1,...,k},{j,7 +1,...,J}], analogously Aj,j.1, Aj-xt,;. 

(A)ij i, j-entry of A. 1.2 

Aij i, j-block in a block matrix of A. 10.1 

A/A[a] Schur complement of Ala]. 4.2, 10.3 

|A| matrix having as entries the absolute values of the entries of matrix A. 9.1 

[Al pa spectral absolute value (A*A)!/?. 17.1 

|A| zero pattern of sign pattern A. 33.2 

VA a square root of a square complex matrix A. 11.2 

Al? the principal square root of a square complex matrix A. 11.2 

Al? the positive semidefinite square root of a positive semidefinite matrix A. 8.3 

A a perturbation A + AA of matrix A. 15.1 (also other meanings) 

||Allx operator norm of matrix A (induced by ||v||,). 37.3 

|| Allez Frobenius (Euclidean) norm of matrix A. 7.1, 37.3 

Alle Ky Fan k norm. 17.3 

| Alls,p Schatten- p norm. 17.3 

||Aller trace norm (not same as Frobenius norm). 17.3 
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unitarily invariant matrix norm. 17.3 

matrix A is positive. 9.1 

matrix A is nonnegative. 9.1 

matrix A is semipositive. 9.1 

A — B is positive. 

A — B is nonnegative. 

A — B is semipositive. 

A is K-nonnegative. 26.1 

A is K -positive. 26.1 

A is K-semipositive. 26.1 

A — B is K-nonnegative. 26.1 

A — B is K-positive. 26.1 

A — B is K-semipositive. 26.1 

matrix A is positive definite. 8.4 

matrix A is positive semidefinite. 8.4 

A — B is positive definite. 8.4 

A — B is positive semidefinite. 8.4 

sign pattern A is a subpattern of sign pattern A. 33.1 

A and B are *-congruent. 8.3 

Aand B are column equivalent (in a domain). 23.2 

Aand B are row equivalent (in a domain). 33.2 

Aand B are equivalent (in a domain). 23.2 

augmented matrix. 1.4 

sum of matrices A and B. 1.2 

matrix product of matrices A and B. 1.2 

direct sum (block diagonal matrix) of matrices A and B. 2.4, 10.2 

Hadamard product of A and B. 8.5 

Kronecker product of A and B. 10.4 

adjacency matrix of graph G, can be shortened to A. 28.3 

adjacency matrix of digraph I, can be shortened to A. 29.2 

the class of all (0, 1)-matrices with row sum vector R and column sum vector 
S. 27.4 

subset of L(V; U) consisting of the antisymmetric maps. 13.5 

a divides b (in a domain). 23.1 

a, b are associates (in a domain). 23.1 

the set of associates (equivalence class) of a (in a domain). 23.1 

any g.c.d. of a),... 5a, (ina domain). 23.1 

the equivalence class of all g.c.d.s of a),... ,a, (in a domain). 23.1 

set complement of a in 1,...,”. Preliminaries 

permutation of real number sequence a, with entries in nonincreasing 
order. Preliminaries 

permutation of real number sequence @, with entries in nondecreasing 
order. Preliminaries 

real number sequence a majorizes §. Preliminaries 

natural number sequence a weakly majorizes 8. Preliminaries 

€-pseudospectral abscissa of complex matrix A. 16.3 

algebraic connectivity of graph G. 36.1 

absolute algebraic connectivity of G. 36.1 

algebraic multiplicity of eigenvalue 4 (of A). 4.3 

max{det W? W|W € {41}"*"}. 32.1 

= a(n,n). 32.1 
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a<p 

adj A 

Alt(v, ® - ++ @ vi) 
Bo 


partial order on partitions of n. 31.10 

adjugate (classical adjoint) of matrix A, also denoted adj(A). 4.2 

= a est sgn(z )Vz(1) @ +++ @ Va(k). 13.6 

zero completion of partial matrix B. 35.6 

basis {v),...,V;,} or ordered basis (v,...,V,) for a vector space. 2.2 
dual basis (for dual space V*) determined by basis B of V. 3.8 
biadjacency matrix of a bipartite graph G, cannot be shortened. 30.1 
number of Jordan blocks with eigenvalue i. 6.2 

vector space of bilinear forms on vector space V over field F. 12.1 
vector space of y-sesquilinear forms (g is an automorphism of F ). 12.4 
biclique of between X, Y C V in graph G = (V, E). 30.3 

max{det W! W|W e {0, 1}*"}. 32.1 

= B(n,n). 32.1 

biclique cover number of graph G. 30.3 

biclique partition number of graph G. 30.3 

complex numbers. Preliminaries 

the open right half plane. Preliminaries 

the closed right half plane. Preliminaries 

the open left half plane. Preliminaries 

the closed left half plane. Preliminaries 

complex conjugate of the complex number c. Preliminaries 

the absolute value of real or complex number c. Preliminaries 

i, jth cofactor of a matrix. 4.1 

cycle on n vertices. 28.1 

n-cycle matrix. 48.1 

graph formed by appending C, to a pendent vertex of P,_,. 36.2 
the number of lines in a minimum line cover of A. 27.1 

the number of 1s in a minimum co-cover of A. 27.1 
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Euclidean length of a column of A (the ith length in nonincreasing or- 


der). 17.1 

kth compound matrix of A. 4.2 

Cauchy matrix. 48.1 

companion matrix of polynomial p(x). 4.3 

convex hull of set S of vectors. Preliminaries 

individual condition number of eigenvalue A. 15.1 
cosine of matrix A. 11.5 

chromatic number of graph G. 28.5 

measure of relative separation of real numbers a, b. 17.4 
an integral domain. 23.1 

a Bezout domain. 23.1 

a greatest common divisor domain. 23.1 

a Euclidean domain. 23.1 

an elementary divisor domain. 23.1 

a principal ideal domain. 23.1 

a unique factorization domain. 23.1 

m X n matrices over domain D. 23.2 

graph formed by n — g isolated vertices to a single vertex of Cy. 36.2 
ring of polynomials p(x) = p(x,,...,X,) with coefficients in D. 23.1 
signed digraph of sign pattern A. 33.1 

digraph of partial matrix B. 35.1 


max{p — q : q vertices may be deleted from graph G leaving p paths}. 34.2 


diag(d),...,dn) 
diam(G) 
dim P 


Ey or E,(A) 
E(A,a + Bi) 
Ex(u) 

eA 

n(G) 

1p (A, bs x) 
End V 
exp(A) 

Fg 
F 


q 
pFmxn 


fF" 

F[x] 

F[x] = F[x, ..., xy] 
F (x) 


FVG 

FAG 

®(G) 

®(S) 

(x) 

fix T 

G 

G orG=(Vo¢, Ec) 
lor! = (Vr, Er) 
G 


Handbook of Linear Algebra 


the partial derivative of F with respect to x. 

the boundary of a set S of real or complex numbers. Preliminaries 

number of edges of a graph having one endpoint inside vertex subset V’ and 

other outside. 28.5 

Kronecker delta, i.e., 1 ifi = j and 0 otherwise. 

the zero part of the inertia of complex square matrix A. 8.3, 19.1 

degree of vertex v in graph G, also denoted 5(v). 28.1 

the set of n x n matrices of nonnegative integers with each row and column 
sum = k, 31.4 

distance from vector v to vector u in a normed vector space. 5.1 

distance from vertex v to vertex u in graph G, also denoted d(v, u). 28.1 

diameter of tree T, same as diam(T). 34.3 

degree of polynomial p(x). 23.1 

determinant of matrix A, also denoted det(A). 4.1 

n x n diagonal matrix with listed diagonal entries. 1.2 

diameter of graph. 28.1 

dimension of convex polytope P, also denoted dim(P). 27.6 

dimension of vector space V, also denoted dim(V). 2.2 

machine precision (i.e., machine epsilon). 37.6 

ith standard basis vector (1 in ith coordinate, 0s elsewhere). 2.1 

standard basis matrix (1 in i, jth coordinate, 0s elsewhere). 2.1 

standard basis e;,...,e, for F”. 2.1 

the identity in the symmetric group S,,. Preliminaries 

the eigenspace for eigenvalue A (of A). 4.3 

the real generalized eigenspace of real matrix A for eigenvalue a + Bi. 6.3 

a particular lower elementary bidiagonal matrix. 21.2 

exponential function of matrix A. 11.3 

a graph parameter involving minimum rank. 28.5 

relative backward error. 38.1 

algebra of linear operators on vector space V. 69.1 

exponential function of matrix A, same as e“. 11.3 

polynomial f(x) divides g(x) in F[x]. 20.2 

finite field with q elements. 61 

m X n matrices over the field F. 1.2 

F -vector space of column n-tuples of elements of F. 1.1 

polynomials over F. 1.1 

ring of polynomials p(x) = p(x,,...,x,) with coefficients in F. 23.1 

field of rational functions in x),...,x, over F. 23.1 

polynomials of degree < n over F. 

all doubly stochastic matrices that have 0’s at least wherever the (0,1)-matrix 
A has 0s. 27.6 

the join ®(F U G) (where (F, G are cone faces). 26.1 

the meet F  G (where (F, G are cone faces). 26.1 

conductance (isoperimetric number) of a graph. 28.5 

face generated by S. 26.1 

face generated by {x}. 26.1 

fixed space of linear transformation T. 3.6 

group. Preliminaries 

graph = (vertices, edges), also denoted G = (V, E). 28.1 

digraph = (vertices, arcs), also denoted T = (V, E). 29.1 

graph complement. 28.1 
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G—xX 
GUH 


y (A) or ya(A) 
A, 

Hn 

H(G) 

H(&2) 


Hy 

I, or I 
B [I] 
ip j 
u(G) 
im(c) 
im(f) 
in(A) = (z, v, 6) 
intK 
ip(A) 
Ja 

Ik 

Jinn 
Tk) 
TRA) 
JCF(A) 
JCFR( A) 
K* 

Ky, 

Kaym 
K;(A,v) 
K(A), kp (A) 
ky (G) 
Ke(G) 
ker A 
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subgraph induced by V \ X. 28.1 

union of graphs G, H. 28.1 

intersection of graphs G, H. 28.1 

join of graphs G, H. 28.1 

(strong) product of graphs G, H. 28.1 

the strong product of @ copies of graph G. 28.1 

the result of deleting vertex v from graph G. 28.1 

a weighted graph. 36.4 

(simple) graph of square matrix A. 19.3, 34.1 

graph of partial matrix B. 35.1 

bipartite graph of sparsity pattern of S. 46.3 

the fill graph, digraph, or bigraph of real square matrix A. 40.5 

bipartite fill-graph of square matrix A. 30.2 

simple graph of convex polytope P. 27.6 

digraph of square matrix A (may have loops). 9.1, 29.2 

the set of all 5-regular graphs on v vertices. 32.7 

Givens transformation. 38.4 

greatest common divisor of (a),..., ax). 

general linear group of order n over F.. 67.1 

the group of D-invertible matrices in D"*". 23.2 

geometric multiplicity of eigenvalue A (of A). 4.3 

Hadamard matrix of size n. 32.2 

the set of Hermitian matrices of size n. 8.1 

the set of Hermitian matrices A such that G(A) = G. 34.1 

the ring of analytic functions over a nonempty path-connected set Q C 
C". 23.1 

= H({¢}). 23.1 

identity matrix or transformation (in F"*" or L(V, V)). 1.2, 3.1 

change-of-basis (transition) matrix from basis B to basis B’. 2.6 

vertex i has access to vertex j (in a digraph). 9.1 

the vertex independence number of graph G. 28.5 

the imaginary part of complex number c. Preliminaries 

image of the function f(= range( f) if f is a linear transformation) 

inertia of complex square matrix A. 8.3, 19.1 

interior of K. 26.1 

the invariant polynomials (i.e., invariant factors) of matrix A. 20.2 

Jordan canonical form of matrix A. 6.2 

real-Jordan canonical form of real matrix A. 6.3 

all 1s matrix in F”™*"”, can be shortened to J, or J when m = n. 

Jordan block of size k for A 6.2 

real-Jordan block of size 2k for A. 6.3 

Jordan canonical form of matrix A. 6.2 

real-Jordan canonical form of matrix A. 6.3 

dual space (dual cone) of cone K. 8.5, 26.1 

complete graph on n vertices. 28.1 

complete bipartite graph on n and m vertices. 28.1 

Krylov subspace of dimension k for matrix A and vector v. 44.1 

condition number of matrix A for linear system Ax = b (in p norm). 37.5 

vertex connectivity of graph G. 36.1 

edge connectivity of graph G. 36.1 

kernel of matrix A, also denoted ker( A). 2.4 


L(Y3..-5 Vin3 U) 
L™(V;U) 

Icm(aj,... 5 4n) 
limo dm = a (C,0) 
limo dm = a (C,k) 
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kernel of linear transformation T, also denoted ker(T). 3.5 

eigenvalue of a matrix or transformation (can also use other Greek). 4.3 

diagonal matrix of eigenvalues. 15.1 

the set of n x n binary matrices in which each row and column sum is k. 31.3 

the Laplacian matrix of graph G. 28.4 

Laplacian matrix of weighted graph G,,. 36.4 

the signless Laplacian matrix of graph G. 28.4 

the line graph of graph G. 28.2 

a generalized line graph. 28.2 

Laguerre polynomial. 31.8 

the set of all linear transformations of V into W. 3.1 

the set of all multilinear maps from V, x--- x V,, into U with operations. 13.1 

L(V,,...5 Vin3 U) with V; = V fori = 1,...,m. 13.5 

least common multiple of a),... 5 dn. 

lim soo Am = a.9.1 

(C, k) limit. 9.1 

principal logarithm of matrix A. 11.4 

the comparison matrix of A. 19.5 

the maximum rank deficiency of graph G. 34.2 

ith eigenvalue of the Laplacian matrix of a graph (nondecreasing order). 28.4 

Colin de Verdiére parameter of graph G. 28.5 

maximum (eigenvalue) multiplicity of graph G. 34.2 

set of matrices A such that bigraph of A is a subgraph of G. 30.3 

the quotient field of H({2). 23.1 

= M({c}). 23.1 

maximal rank of sign pattern A. 33.6 

minimal rank of sign pattern A. 33.6 

minimum rank of graph G. 34.2 

natural numbers, i.e., {0,1,2,...}. 

the set of integers {1,2,...,n}. 9.1 

the kth generalized eigenspace of square matrix A. 6.1 

the generalized eigenspace of matrix A, can be abbreviated N, (A). 6.1 

the minimum number of distinct eigenvalues of a matrix in S(T) where T is 
a tree. 34.3 

the negative part of the inertia of complex square matrix A. 8.3, 19.1 

the index of matrix A at eigenvalue A. 6.1 

the index of P, i.e. vp(p). 9.3 

the co-index of P. 9.3 

the (vertex-edge) incidence matrix of graph G. 28.4 

the number of nonzero entries in sparse matrix A. 40.2 

nullity of a matrix, also null(A). 2.4 

nullity of a linear transformation or of a matrix, also null(T). 3.5 

big-oh of function f. Preliminaries 

little-oh of function f. Preliminaries 

set of n x n doubly stochastic matrices. 9.4 

clique number of graph G. 28.4 

path on n vertices. 28.1 

principal submatrix at a distinguished eigenvalue. 9.3 

path cover number of graph G. 34.2 

tree constructed in a specific way. 36.3 

the positive part of the inertia of complex square matrix A. 8.3, 19.1 
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TI, 
pa(x) 
Pa(x) 
PD, 

per A 
per, A 
Projy(v) 
Proj w,z(v) 
PSD, 

Q 

Q(A) 
qa(x) 
Qn 


QR 

OR 

p(A) 

pe(A) 

Q(A) 

o*(A) 

p(G) 

pi(A, x), p2(A, x) 
R 


REF(A) 
RREF(A) 
reldist(H, H) 
RS(A) 

S4 

S\ X 
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regular n—sided unit polygon. 20.3 

characteristic polynomial of matrix A, or linear transformation. 4.3 

characteristic polynomial of graph G. 28.3 

the set of n x n positive definite matrices, can be shortened to PD. 8.4 

permanent of matrix A, also denoted per(A). 31.1 

the sum of the subpermanents of order k, also denoted per; (A). 31.1 

the orthogonal projection of v onto W (along W*). 5.4 

the projection of v onto W along Z. 3.6 

the set of n x n positive semidefinite matrices, can be shortened to PSD. 8.4 

rational numbers. 

qualitative class of sign pattern A (or of sgn(A) if A is a real matrix). 33.1 

minimum polynomial of matrix A, or linear transformation. 4.3 

the set of all strictly increasing sequences of r integers chosen from the set 
{1,2,...,n}. 23.2 

QR-factorization of a real or complex matrix. 5.5 

reduced QR-factorization of a real or complex matrix. 5.5 

spectral radius of real or complex square matrix A. 4.3 

€-pseudospectral radius of complex matrix A. 16.3 

term rank of matrix A. 27.1 

largest size of a zero submatrix of A. 27.1 

representation of group G. 68.1 

rook polynomials. 31.8 

real numbers. 

positive reals (subset of R or C). 

nonnegative reals (subset of R or C). 

negative reals (subset of R or C). 

nonpositive reals (subset of R or C). 

max-plus semiring. 25.1 

completed max-plus semiring. 25.1 

reduced digraph of I’. 9.1, 29.5 

basic reduced digraph of matrix P. 9.3 

polynomials of degree < n over R. 

Euclidean length of a row of A (the ith length in nonincreasing order). 17.1 

elementary divisors rational canonical form of A. 6.4 

invariant factors rational canonical form of A. 6.6 

the real part of complex number. c 

range of matrix A, = column space of A, also denoted range(A). 2.4 

range of linear transformation T, also denoted range(T). 3.5 

rank of matrix A, also denoted rank(A). 1.3, 2.4 

rank of bilinear form f. 12.1 

rank of linear transformation T, also denoted rank(T). 3.5 

a row echelon form of matrix A. 1.3 

the reduced row echelon form of matrix A. 1.3 

component-wise relative distance between H, H. 46.4 

row space of matrix A. 2.4 

annihilator (in the dual space V*) of subspace S of vector space V. 3.8 

set complement of X in S. Preliminaries 

the symmetric group on {1,...,/}. Preliminaries 

the set of real symmetric matrices of size n. 8.1 

ith singular value (in nonincreasing order). 5.6 

spectrum of square matrix A. 4.3 
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a(G) 
0-(A) 

oF (A) 
o,(A, B) 
o,(A, B,C) 
o-(P) 

|S] 

Sk(A) 
S(G) 
S™(V;U) 
Sx (O15... 5 Qn) 
S(u) 
sep(A,, Az) 
sgn(z) 
sgn(a) 
sgn(A) 
sgn(a) 
sign(A) 
sign(z) 


SL(n, F) 
SMR(A) 
smr(A) 
Span(S) 
Struct(A) 
sv(A) 
SVext( A) 


Sym(v, @ --- @ vk) 


T 

T7! 
Tr 
BIT] 
IT] 
[T] 


Ta 
W Bi 
T(k,1,d) 


(A) 
T(R) 
B(G) 
@(G) 
O(X, Y) 
trA 
Uu(T) 


UxV* 
UxVe 
V 

vr 
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spectrum of graph G. 28.3 

€-pseudospectrum of complex square matrix A. 16.1 

real structured ¢-pseudospectrum of real square matrix A. 16.5 
€-pseudospectrum of the matrix pencil A — xB. 16.5 

spectral value set of the matrix triplet A, B, C. 16.5 
€-pseudospectrum of matrix polynomial P. 16.5 

cardinality of set S. 

sum of all principal minors of size k of matrix A. 4.2 

the set of real symmetric matrices A such that G(A) = G. 34.1 
subset of L(V; U) consisting of the symmetric maps. 13.5 

kth elementary symmetric function of a;,i = 1,...,/. Preliminaries 
rev (d(u, k))* where G = (V, E) is a simple graph. 36.5 

separation between matrices Aj, A2. 15.1 

sign of the permutation z. Preliminaries 

sign of the real number a as used in sign patterns, one of +, 0, —. Preliminaries 
sign pattern matrix of the real matrix A, with entries in +, 0, —. 33.1 
sign of a set a of edges in a signed bipartite graph. 30.1 

matrix sign function of the complex square matrix A. 11.6 

sign of the complex number z as used in numerical linear algebra, always 
nonzero. Preliminaries 

special linear group of order n over F. 67.1 

symmetric maximal rank of sign pattern A. 33.6 

symmetric minimal rank of sign pattern A. 33.6 

span of the set S of vectors. 2.1 

the sparsity structure (support) of A. 9.6, 40.3 

vector of singular values of complex matrix A. 15.2, 17.1 

extended vector of singular values of complex matrix A. 15.2 

= 5) Uwes, Vall) @ +++ ® Vuk). 16.6 

linear transformation T. 3.1 

inverse of linear transformation T. 3.7 

transpose of linear transformation T. 3.8 

matrix of T with respect to bases B (input) and B’ (output). 3.3 
matrix of T with respect to bases B and B (same as g[T]g). 3.3 

for T:F” — F"™, matrix of T with respect to the standard bases, [T] = 


€,[T le, ). 3.3 
linear transformation associated to matrix A, T,(v) = Av. 3.3 
triples of subsets of {1,... ,} of cardinality r. 17.6 


tree constructed from Pz by appending k and / isolated vertices to the 
ends. 36.3 

the ergodicity coefficient of matrix A. 9.1 

the class of all tournament matrices with score vector R. 27.5 

Lovasz parameter of graph G. 28.5 

Shannon capacity of graph G. 28.5 

canonical angle matrix between X and Y. 15.1 

trace of matrix A, also denoted tr(A). 1.2 

the minimum number of simple eigenvalues of a matrix in S(T) where T is 
a tree. 34.4 

singular value decomposition of a real or complex matrix. 5.6 

reduced singular value decomposition of a real or complex matrix. 5.6 

vector space V. 1.1 

dual space of V. 3.8 


Notation Index N-9 


Ve bidual space of V (dual of the dual). 3.8 

Vix xX Vy external direct sum of vector spaces V;. 2.3 

Vi @:-:-@V; tensor product of vector spaces V;. 13.2 

@Q”"V V @---@ V (mcopies of V). 13.2 

®V the tensor algebra on V . 13.9 

V"V the symmetric space of degree m. 13.6 

VV the symmetric algebra on V. 13.9 

A" V the Grassman (exterior) space of degree m. 13.6 

AV the Grassman algebra on V. 13.9 

Vw an (n—1) x (n—1) (0, 1)-matrix constructed from ann x n +1-matrix. 32.2 
V/W quotient space of V by subspace W. 2.3 

Vv vector v. 1.1 

Vv a perturbation v + Av of vector v. 15.1 (Also other meanings.) 

(v, u) inner product of vectors v and u. 5.1 

lvl norm of vector v (which norm depends on context). 5.1, 37.1 

lIvll2 Euclidean norm of vector v in R" or C”, = standard inner product norm. 5.1, 37.1 
IIVlloo oo - norm (maximum of absolute values) of vector v in R” or C". 37.1 
IIvlla 1-norm (absolute column sum) of vector v in R" or C”. 37.1 

Ilvlluz unitarily invariant norm. 15 

[v]z coordinate vector of v with respect to basis B. 2.6 

vilw v is orthogonal to w. 5.2 

Vi @-:- Qvy tensor product of vectors v;. 13.2 

ViVi VME symmetric product of vectors v;. 13.6 

Vi Art AVE exterior product of vectors v;. 13.6 

v>0 v is nonnegative. 9.1 

v>0 vis positive. 9.1 

v>0 v is semi-positive. 9.1 

v>w v — wis nonnegative. 

v>w v — wis positive. 

v>w v — wis semi-positive. 

v>*ko vis K -nonnegative. 26.1 

v>* 0 v is K -positive. 26.1 

v>*ko vis K -semipositive. 26.1 

v>k w v — wis K -nonnegative. 26.1 

v>k w v — wis K-positive. 26.1 

v>k w v — wis K -semipositive. 26.1 

vec A the vector of columns of A. 10.4 

wt orthogonal complement of subspace W. 5.2 

WiOd--- OW direct sum of subspaces W;. 2.3 

Wi +--- + W, sum of subspaces Wj. 2.3 

Wy an(n+1)x(n+1) +1-matrix constructed from ann x n (0, 1)-matrix. 32.2 
W(A) numerical range of A. 7.1, 18.1 

w(A) numerical radius of complex square matrix A. 18.1 

w(A) distance of W(A) to the origin. 18.1 

X characteristic vector of X C V where G = (V, E). 30.3 

XLS least squares solution. 39.1 

Z(x%y) the canonical angle between the two vectors x, y; = ©({x}, {y}). 15.1 
Z integers. 

Zt positive integers. 

Zy integers mod n. 23.1 


Zn n x n lower shift matrix. 48.1 
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Abelian, Lie algebras, 70-2 
Abs, Mathematica software, 73-26 
Absolute, simple graphs, 36-9 to 36-10 
Absolute bound, 28-11 
Absolute errors 
conditioning and condition numbers, 37-7 
floating point numbers, 37-13, 37-16 
Absolute irreducibility, 67-1 
Absolute matrix norms, 37—4 
Absolute values, 17-1 
Absolute vector norm, 37—2 
Absorbing 
irreducible classes, 54—5 
vector norms, 37-3 
vector seminorms, 37—4 
Access equivalence 
irreducible classes, 54—5 
irreducible matrices, 29-6, 29-7 
max-plus eigenproblem, 25-6 
nonnegative and stochastic matrices, 9-2 
Action, group representations, 68-2 
Active branch, algebraic connectivity, 36-5 
Active constraints, 51-1 
Acyclic matrices 
multiplicative D-stability, 19-6 
rank revealing decompositions, 46-9 
Adaptive filtering, signal processing, 64-12 to 
64-13 
Addition, 1-1, 1-3 
Additive coset, 2-5 
Additive D-stability, 19-7 to 19-8 
Additive identity axiom, 1-1 
Additive IEPs (AIEPs), 20-10 
Additive inverse axiom, 1-1 
Additive preservers, 22—7 to 22-8 
Adjacency 
convex set points, 50-13 
digraphs, 29-3 to 29-4 
graphs, 28-5 to 28-7 
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Hermitian matrices, 8—2 
linear preservers, 22-7 
vertices, 28-2 

Adjoint linear transformation, 51-3 

Adjoint map, Lie algebras, 70-2 

Adjoints, inner product spaces, 13-22 

Adjoints of linear operators 
inner product spaces, 5-5 to 5-6 
semidefinite programming, 51-3 

Adjugates, determinants, 4—3 

Adjusting random vectors, 52-4 

Admissibility, control theory, 57-2 

Admittance matrix, 28-7 

Ad-nilpotency, 70-4 

Ad-semisimple linear transformation, 70—4 

Advanced linear algebra 
bilinear forms, 12-1 to 12-9 
cone invariant departure, matrices, 26-1 to 26-14 
equalities, matrices, 14—1 to 14-17 
functions of matrices, 11-1 to 11-12 
inequalities, matrices, 14-1 to 14-17 
inertia, matrices, 19-1 to 19-10 
integral domains, matrices over, 23-1 to 23-10 
inverse eigenvalue problems, 20-1 to 20-12 
linear preserver problems, 22—1 to 22-8 
matrix equalities and inequalities, 14-1 to 14-17 
matrix perturbation theory, 15-1 to 15-16 
matrix stability and inertia, 19-1 to 19-10 
max-plus algebra, 25-1 to 25-14 
multilinear algebra, 13-1 to 13-26 
numerical range, 18-1 to 18-11 
perturbation theory, matrices, 15-1 to 15-16 
pseudospectra, 16-1 to 16-15 
quadratic forms, 12-1 to 12-9 
sesquilinear forms, 12—1 to 12-9 
similarity of matrix families, 23-1 to 23-10 
singular values and singular value inequalities, 17-1 

to 17-15 

stability, matrices, 19-1 to 19-10 
total negativity, matrices, 21-1 to 21-12 
total positivity, matrices, 21-1 to 21-12 
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Affine algebraic variety, 24-8 
Affine function, 50-1 
Affine-independence, 65-2 
Affine parameterized IEPs (PIEPs), 20-10 to 20-12 
Affine subspace determination, 65-2 
AIEPs (additive IEPs), 20-10 
Aissen studies, 21-12 
Aitken estimator, 52-8 
Akian, Marianne, 25-1 to 25-14 
Akivis algebra, 69-16 to 69-17 
Albert algebra, 69-4, 69-14 
Alexandroff inequality, 25-10 
Alexandrov’s inequality, 31-2 
Algebra, P-1 
Algebra applications, see also Nonassociative algebra 
group representations, 68-1 to 68-11 
Lie algebras, 70-1 to 70-10 
matrix groups, 67-1 to 67-7 
nonassociative algebra, 69-1 to 69-25 
Algebraic aspects, least squares solutions, 39-4 to 39-5 
Algebraic connectivity 
absolute, simple graphs, 36-9 to 36-10 
Fiedler vectors, 36—7 to 36-9 
generalized Laplacian, 36-10 to 36-11 
matrix representations, 28-7 
multiplicity, 36-10 to 36-11 
simple graphs, 36—1 to 36-4, 36-9 to 36-10 
trees, 36—4 to 36-6 
weighted graphs, 36—7 to 36-9 
Algebraic eigenvalues, 25-9, see also Eigenvalues and 
eigenvectors 
gebraic function, matrix similarities, 24-1 
gebraic geometric Goppa (AG) code, 61-10 
gebraic multigrid, preconditioners, 41-11 
gebraic multiplicity, 4-6 
gebraic Riccati equation (ARE), 57-10, 57-12 
gorithms, see also specific algorithm 
Arbitrary Precision Approximating (APA), 47-6 
Arnoldi, 41-7 
BiCGSTAB, 41-8 
biconjugate gradient (BCG/BiCG), 41-7, 49-13 
bilinear noncommutative, 47-2 
bit flipping algorithm, 61-11 
Conjugate Gradient (CG), 41-4, 41-6 
conjugate gradient squared (CGS), 41-8 
Denardo, 25-8 
error analysis, 37-16 to 37-17 
ESPRIT, 64-17 
Euclid’s, 23-2 
fast matrix multiplication, 47-2 to 47-7 
Full Orthogonalization Method (FOM), 41-7 
Generalized Minimal Residual (GMRES), 41-7, 49-13 
Lanczos algorithm, 41-4 to 41-5 
least squares solutions, 39-6 to 39-7 
left-preconditioned BiCGSTAB algorithm, 41-12 
Levinson-Durbin algorithm, 64-8 
Minimal Residual (MINRES), 41—4, 41-6 
MUSIC, 64-17 
noncommutative, 47—2 
non-Hermitian Lanczos algorithm, 41-7 
policy iteration, 25-7 
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power algorithm, 25-7 
preconditioned conjugate gradient (PCG), 41-13 
quasi-minimal residual (QMR), 41-8, 49-13 
restarted GMRES algorithm, 41-7 
singular value decomposition, 45-4 to 45-12 
transpose-free quasi-minimal residual (TFQMR), 
49-14 
two-sided Lanczos algorithm, 41-7 
All-ones matrix, 52—4 
Allowing characteristics 
random vectors, 52-4 
sign-pattern matrices, 33-9 to 33-11 
Alphabet, coding theory, 61-1 
Alternate path, single arc, 35-14 
Alternating bilinear forms, 12-5 to 12-6 
Alternative algebras, 69-2, 69-10 to 69-12 
Alternative bimodule, 69-10 
Alternator, 13-12 
Alt multiplication, 13-17 to 13-19 
AMG code, 41-12 
Analysis applications 
control theory, 57—1 to 57-17 
differential equations, 55-1 to 55-16 
dynamical systems, 56-1 to 56-21 
Fourier analysis, 58-1 to 58-20 
LTI systems, 57—7 to 57-10 
stability, 55-1 to 55-16 
Analytical similarity, 24—1 
Analyzing fill, 40-10 to 40-13 
Angles, inner product spaces, 5-1 
Angular momentum, 59-9 to 59-10 
Annihilator, 3—8 to 3-9 
Anticommutative algebra, 69-2 
Anticommutativity, 70-2 
Anticommutator, 69-3 
Anti-identity matrices, 48-2 
Antisymmetric maps, 13-10 to 13-12 
Antisymmetry, 12-5 
Aperiodicity 
characterizing, 9-3 
irreducible classes, 54—5 
irreducible matrices, 9-3 
reducible matrices, 9-7 
Append, Mathematica software 
linear systems, 73-23 
matrices manipulation, 73-13 
vectors, 73-3 
AppendColumns, Mathematica software, 73-13 
Appending, vertices, 36-3 
AppendRows, Mathematica software, 73-13 
Applications 
fast matrix multiplication, 47-9 to 47-10 
semidefinite programming, 51-9 to 51-11 
Applications, algebra 
group representations, 68-1 to 68-11 
Lie algebras, 70-1 to 70-10 
matrix groups, 67—1 to 67-7 
nonassociative algebra, 69-1 to 69-25 
Applications, analysis 
control theory, 57-1 to 57-17 
differential equations, 55-1 to 55-16 
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dynamical systems, 56—1 to 56-21 
Fourier analysis, 58-1 to 58-20 
stability, 55-1 to 55-16 
Applications, biological sciences, 60—1 to 60-13 
Applications, computer science 
coding theory, 61-1 to 61-13 
information retrieval, 63—1 to 63-14 
quantum computation, 62-1 to 62-19 
signal processing, 64—1 to 64-18 
Web searches, 63—1 to 63-14 
Applications, geometry 
Euclidean geometry, 66-1 to 66-15 
geometry, 65-1 to 65-9 
Applications, optimization 
linear programming, 50-1 to 50-24 
semidefinite programming, 51-1 to 51-11 
Applications, physical sciences, 59-1 to 59-11 
Applications, probability and statistics 
linear statistical models, 52-1 to 52-15 
Markov chains, 54-1 to 54-14 
multivariate statistical analysis, 53-1 to 53-14 
random vectors, 52—1 to 52-15 
Apply, Mathematica software, 73-3, 73-5, 73-27 
Approximate Jordan form, Maple software, 72-15 
Approximate prescribed-line-sum scalings, 9-21 to 
9-22 
Approximation 
fast matrix multiplication, 47-6 to 47-7 
linear programming, 50-20 to 50-23 
orthogonal projection, 5-7 
Arbitrary Precision Approximating (APA) algorithms, 
47-6 
ArcSin, Mathematica software, 73-26 
ARE, see Algebraic Riccati equation (ARE) 
Arithmetic Euclidean vector space, 66-1 
Arm, stars, 34-10 
Arnold Hypothesis, Strong, 28-9, 28-10 
Arnoldi algorithm, 41-7, 41-8 
Arnoldi decomposition, 16-11 
Arnoldi factorization 
implicitly restarted Arnoldi method, 44-2 to 44-4 
pseudospectra, 16-3 
Arnoldi matrices, 49-11 
Arnoldi method, see also Implicitly restarted Arnoldi 
method (IRAM) 
eigenvalue computations, 49-12 
implicit restarting, 44-6 
large-scale matrix computations, 49-10 to 49-11 
sparse matrices, 43-9, 43-11 
Arnoldi vectors 
Arnoldi factorization, 44—3 
Arnoldi process, 49-11 
ARPACK subroutine package 
computational modes, 76-8 to 76-9 
directory structure and contents, 76-3 
fundamentals, 76—1 to 76-2, 76—4 to 76—5 
Lanczos methods, 42-21 
Matlab’s EIGS, 76-9 to 76-10 
naming conventions, 76-3 to 76-4 
precisions, 76-3 to 76—4 
pseudospectra computation, 16-12 


reverse communication, 76-2 

setup of problem, 76-5 to 76-7 

sparse matrices, 43-9 

types, 76-3 to 76-4 

use, 76—7 to 76-8 
Array, Maple software, 72-1, 72-2 
array, Maple software, 72—1 
Array, Mathematica software 

matrices, 73-6 

vectors, 73-3, 73-4 
Array arithmetic, Maple software, 72-8 
ArrayPlot, Mathematica software 

fundamentals, 73-27 

matrices, 73-7, 73-9 
ArrayRules, Mathematica software, 73-6 
Arrays 

Fourier analysis, 58-2 

manifold, 64-16 

Maple software, 72-8 to 72-9 

response vector, 64-16 
Arrays, Maple software, 72-8, 72-9 
Arrival estimation direction, 64-15 to 64-18 
Arrow, Mathematica software, 73-5 
Artin’s theorem, 69-3, 69-11 
AspectRatio, Mathematica software, 73-27 
Assignment polytope, 27-10 
Associated divisor, 61-10 
Associated linear programming, 50-14 
Associated maps, 13-19 to 13-20 
Associations, generalized stars, 34-11 
Association schemes, graphs, 28-11 to 28-12 
Associative algebra, 69-2 
Associative center, 69—5 
Associative enveloping algebra, 69-13 
Associative nucleus, 69-5 
Associativity axiom, 1-1, 1-2 
Associator, nonassociative algebra, 69-2 
Associator ideal, 69-5 
Asymmetric digraphs, 35-2, see also Digraphs 
Asymptotics, matrix powers, 25-8 to 25-9 
Asymptotic spectrum, 47-8 
Asymptotic stability 

control theory, 57-2 

linear differential-algebraic equations, 55-14 

linear ordinary differential equations, 55-10 

LTI systems, 57—7 
ATLAST M-file collection, 71-20 
Attractor-repeller decompositions, 56-7 
Augmented matrices 

Bezout domains, 23-9 

systems of linear equations, 1-9, 1-11 
Augmented systems, least squares solution, 39-4 
Autocorrelation matrix, 64-5 
Autocorrelation sequence, 64—4 
Automatic, Mathematica software 

fundamentals, 73-27 

singular values, 73-17 
Automorphism 

Lie algebras, 70-1 

ring, linear preservers, 22-7 
Autonomy, control theory, 57-2 


1-3 


1-4 


B 


Backward errors 
analysis, 37-20 
numerical stability and instability, 37-18, 37-20 to 
37-21 
one-sided Jacobi SVD algorithm, 46-2 
BackwardsSubstitute, Maple software, 72-9 
Backward stability 
numerical stability and instability, 37-18 
one-sided Jacobi SVD algorithm, 46-2 
Bad columns, 50-8 
Badly-conditioned data, 37—7 
Bad rows, 50-8 
Bai, Zhaojun, 75—1 to 75-23 
Bailey, D., 47-5 
Balanced Boolean function 
Deutsch-Jozsa problem, 62-9 
Deutsch’s problem, 62-8 
Balanced column signing, 33-5 
Balanced matrices 
D-optimal matrices, 32-12 
nonsquare case, 32—5 
Balanced row signing, 33-5 
Balanced vectors, 33-5 
Banded matrices, 41-2 
Banding, Toeplitz matrices, 16-6 
Bandlimited random signals, 64-5 
Bapat, Ravindra, 25-1 to 25-14 
Barely L-matrices, 33-5 
Barioli, Francesco, 3-1 to 3-9 
Barker and Schneider studies, 26-3 
Barrett, Wayne, 8—1 to 8-12 
Barvinok rank, 25-13 
Barycentric coordinates, 66-8 
Bases 
Bezout domains, 23-8 
complex sign and ray patterns, 33-14 
component, linear skew product flows, 56-11 
coordinates, 2-10 to 2-12 
induced, symmetric and Grassmann tensors, 13-15 
LTI systems, 57—7 
orthogonality, 5-3 
semisimple and simple algebras, 70-4 
similarity, 3-4 
vector spaces, 2—3 to 2-4 
Basic class of P, 9-7 
Basic variable, 1-10 
Bauer-Fike theorem 
eigenvalue problems, 15-2 
pseudospectra, 16-2 to 16-3 
BCG/BiCG (biconjugate gradient) algorithm 
Krylov space methods, 41-7, 41-10 
linear systems of equations, 49-13 
preconditioners, 41-12 
BCH (Bose-Chadhuri-Hocquenghem) code, 61-8, 
61-9 to 61-10 
Beattie, Christopher, 38-1 to 38-17 
Belitskii reduction, 24-10 
Benner, Peter, 57—1 to 57-17 
Bernoulli random variables, 52-2 
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Bernstein-Vazirani problem, 62-11 to 62-13 

Bessel’s Inequality, 5—4 

Best Approximation Theorem, 5-7 

Best linear approximation, 50-20 

Best linear unbiased estimate (BLUE), 39-2 

Best linear unbiased estimator, 52-9 

Best PSD approximation, 17-13 

Best rank k approximation, 17-12 to 17-13 

Best unitary approximation, 17-13 

Between-groups matrix, 53-6 

Bezout domains 
certain integral domains, 23-2 
matrices over integral domains, 23-8 to 23-9 
matrix equivalence, 23-6 

Bhatia studies, 17-13 

Biacyclic matrices, 46-8 

Biadjacency matrix, 30-1 

Biclique, bipartite graphs, 30-8 

Biclique cover, 30-8 

Biclique cover number, 30-8 

Biclique partition, 30-8 

Biclique partition number, 30-8 

Biconjugate gradient (BCG/BiCG) algorithm 
Krylov space methods, 41-7, 41-10 
linear systems of equations, 49-13 
preconditioners, 41-12 

Bideterminants, 25-13 

Bidiagonal singular values by bisection, 45-9 

Bidual space, 3-8 

Bigraphs, 40-10, 40-11 to 40-12, see also Bipartite 

graphs 

Bilinear forms 
alternating forms, 12-5 to 12-6 
fundamentals, 12-1 to 12-3 
Jordan algebras, 69-13 
symmetric form, 12-1 to 12-5 

Bilinear maps, 13-1 

Bilinear noncommutative algorithms, 47—2 

Bimodule algebras, 69-6 

Binary even weight codes, 61—4 

Binary Golay code, 61-8, 61-9 

Binary linear block code (BLBC), 61-3 

Binary matrices, permanents, 31-5 to 31-7 

Binary symmetric channel, 61-3 to 61-4 

Binary trees, 34-15 

Binet-Cauchy identity, 25-13 

Binet-Cauchy theorem, 31-2 

Bini, Dario A., 47—1 to 47-10 

Binomial, Mathematica software, 73-26 

Binomial distribution, 52—4 

Biological sciences applications, 60-1 to 60-13 

Biomolecular modeling 
flux balancing equation, 60-10 to 60-13 
fundamentals, 60-1, 60-13 
Karle-Hauptman matrix, 60-7 to 60-9 
mapping, 60-2 to 60-4 
metabolic network simulation, 60-10 to 60-13 
NMR protein structure determination, 60-2 to 60-4 
protein motion modes, 60—9 to 60-10 
protein structure comparison, 60-4 to 60-7 
x-ray crystallography, 60-7 to 60-9 
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Biorthogonalization singular value decomposition, 
45-11 
Biorthogonal pair of bases, 66-5 
Bipartite graphs 
factorizations, 30-8 to 30-10 
fill-graph, 30-4 
fundamentals, 30-1 to 30-3 
graphs, 28-2 
matrices, 30—4 to 30-7 
modeling and analyzing fill, 40-10 
rank revealing decomposition, 46-8 
Bipartite sign pattern matrices, 33-9 
Birepresentations, 69-6 
Birkhoff’s theorem, 27-11 
Bisection method, 42-14 to 42-15 
Bit flipping algorithm, 61-11 
Bit quantum gate, 62-2 
Black-box, 66-13 
Black-box matrix, 66-13 to 66-15 
Bland’s rule, 50-12, 50-13 
BLAS subroutine package 
fundamentals, 74-1 to 74-7 
method comparison, 42-21 
BLBC (binary linear block code), 61-3 
ock, graphs and digraphs, 35-2 
ock-clique, 35-2 
ock code of length, 61-1 
ock diagonal matrices, 10—4 to 10-6 
ock lower triangular matrices, 10-4 
ock matrices 
partitioned matrices, 10-1 to 10-3 
structured matrices, 48-3 
ockMatrix, Mathematica software, 73-13 
ock positive semidefinite matrices, 17—9 
ocks, square case, 32-2 
ock-Toeplitz matrices, 48-3 
ock-Toeplitz-Toeplitz-Block (BTTB) matrices, 
48-3 
Block triangular matrices 
inequalities, 17-9 
partitioned matrices, 10-4 to 10-6 
Block upper triangular matrices, 10—4 
Bloomfield-Watson efficiency, 52-9, 52-10, 52-13 to 
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52-14 

Bloomfield-Watson-Knott Inequality, 52-10, 52-13 
to 52-14 

Bloomfield-Watson Trace Inequality, 52-10, 52-13 
to 52-14 


BLUE (best linear unbiased estimate), 39-2 
BN structure, matrix groups, 67—4 to 67-5 
Bochner’s theorem, 8-10, 8-11, 8-12 
Boolean properties 
algebra, 30-8 
bipartite graphs, 30-8, 30-9 
Deutsch-Jozsa problem, 62-9 
Deutsch’s problem, 62-8 
fast matrix multiplication, 47-10 
matrices, 30-8, 47-10 
rank, 30-8, 30-9 
Borel subgroup, 67—4, 67-5 
Borobia, Alberto, 20-1 to 20-12 


Bose-Chadhuri-Hocquenghem (BCH) code, 61-8 
Bose-Mesner algebra, 28-11 
Bosons, 59-10 
Bottleneck matrices, 36—4 
Boundaries 
fundamentals, P-1 
interior point methods, 50-23 
numerical range, 18-3 to 18-4 
Bounded properties, 9-11 
Box, Euclidean spaces, 66-10 
Branches 
matrix similarities, 24—1 
multiplicities and parter vertices, 34-2 
Brauer theorem, 14—6 
Brégman’s bound, 31-7 
Bremmer, Murray R., 69-1 to 69-25 
Brent, R., 47-5 
Brin, Sergey, 54—4, 63-9, 63-10 
Browne’s theorem, 14—2 
Brualdi, Richard A., 27-1 to 27-12 
BTTB (Block-Toeplitz-Toeplitz-Block) matrices, 
A8-3 
bucky command, Matlab software, 71-11 
Built-in functions, Matlab software, 71-4 to 71-5 
Bulge, 42-10, 43-5 
Bunch-Parlett factorization, 46-14 
Burnside’s Vanishing theorem, 68-6 
Businger-Golub pivoting 
preconditioned Jacobi SVD algorithm, 46-5 
rank revealing decompositions, 46-9 
Butterfly relations, 58-18, 58-20 
Byers, Ralph, 37-1 to 37-21 
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Cameron, Peter J., 67—1 to 67—7 
Canonical angle and canonical angle matrix, 15-2 
Canonical angles, 17-15 
Canonical correlations and variates 
multivariate statistical analysis, 53-7 
singular values, 17-15 
Canonical forms 
eigenvectors, generalized, 6-2 to 6-3 
elementary divisors, 6-8 to 6-11 
fundamentals, 6—1 to 6-2 
invariant factors, 6-12 to 6-14 
Jordan canonical form, 6—3 to 6-6 
linear programming, 50-7, 50-7 to 50-8 
Maple software, 72-15 to 72-16 
rational canonical forms, 6-8 to 6-11, 6-12 to 
6-14 
real-Jordan canonical form, 6-6 to 6-8 
Smith normal form, 6—11 to 6-12 
Canonical variates, 53-6 
Cartan matrix, 70—4 
Cartan’s Criterion for Semisimplicity, 70-4 
Cartesian coordinates, 55-3 
Cartesian decomposition, 17-11 
Cartesian product, 13-11 to 13-12 
Cassini, ovals of, 14-6, 14-6 to 14—7 
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cat command, Matlab software, 71-2 
Cauchy-Binet formula 
determinants, 4—4, 4-5 
matrix equivalence, 23-6 
Cauchy-Binet Identity, 21-2 
Cauchy-Binet inequalities, 14-10 
Cauchy boundary conditions, 55-3 
Cauchy integral, 11-2 
Cauchy interlace property, 42-20 
Cauchy matrices 
rank revealing decompositions, 46-9 
structured matrices, 48-2 
symmetric indefinite matrices, 46-16 
totally positive and negative matrices, 21-4 
Cauchy-Schwartz inequality 
inner product spaces, 5—1, 5-2 
vector norms, 37-3 
Causal part, Wiener filtering, 64-10 
Causal signal processing, 64—2 
Cayley-Dickson algebra 
alternative algebra, 69-11, 69-12 
Jordan algebras, 69-16 
Malcev algebras, 69-17 
nonassociative algebra, 69-4 
power associative algebras, 69-15 
Cayley-Dickson doubling process, 69-4 
Cayley-Dickson matrix algebra, 69-9 
Cayley-Hamilton Theorem, 4-8 
Cayley numbers 
nonassociative algebra, 69-4 
standard forms, 22—4 
Cayley’s Formula, 7-6 
Cayley’s Transform, 7-6 
CCS (compressed column storage) scheme, 40-3 
Cell bracket, Mathematica software, 73-2 
Cells, Mathematica software, 73-2 
Censoring, Markov chains, 54-11 
Center, 69-5 
Centering matrix, 52-4 
Center subspaces, 56-3 
Central algebra, 69-5 
Central controller, 57-15 
Central distribution, 53-3 
Central force motion, 59-4 to 59-5 
Central matrices, 33-17 
Central Moufang identity, 69-10 
Central path, 51-8 
Central vertex, 34-10 
Centroid, 69-5 
Certain integral domains, 23-1 to 23-4 
CG (Conjugate Gradient) algorithm 
convergence rates, 41-14 to 41-15 
Krylov space methods, 41-4, 41-6 
CGS (classical Gram-Schmidt) scheme, 44-3 to 
44—4, see also Gram-Schmidt methods 
CGS (Conjugate Gradient Squared) algorithm, 
41-8 
Chain exponent, 56-16 
Chain recurrence, 56—7 to 56-9 
Chain recurrent component, 56-7 
Chain recurrent set, 56—7 
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Chain transitive, 56-7 
Change of basis 


coordinates, 2-10 to 2-12 

LTI systems, 57-7 

similarity, 3-4 

hange-of-basis matrix, 2-10 

hannels, coding theory, 61-2 
haracteristic equation, 43-2 
haracteristic polynomial 

adjacency matrix, 28-5 

fast matrix multiplication, 47-10 
generalized eigenvalue problem, 43-2 


haracteristicPolynomial, Maple software 

eigenvalues and eigenvectors, 72-11, 72-12 

matrix stability, 72-21 

nonlinear algebra, 72-14 

haracteristic polynomial function, 25-9 

haracteristic polynomials, 4-6 

haracteristicPolynomials, Mathematica software, 
73-14 

haracteristic vector, 30-8 

haracteristic vertex, 36—4 

haracterizations, singular values, 17-1 to 17-3 


haracters 

grading, 70-9 

group representations, 68-5 to 68-6 
restriction, 68—8 to 68-10 

table, 68-6 to 68-8 


Character table, 68—7 
Chebyshev polynomial 


convergence rates, 41-15 

polynomial restarting, 44-6 

rook polynomials, 31-11 

heckerboard partial order, 21-9 

hemical flux, 60-10 

holesky algorithm, 46-11 

holesky decomposition 
preconditioners, 41-12, 41-13 
symmetric factorizations, 38-15, 38-16 
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holesky factor 

modeling and analyzing fill, 40-13 
positive definite matrices, 46-12 
QR factorization, 39-9 

reordering effect, 40-16, 40-18 


Cholesky factorization 


extensions, 16-13 

least squares algorithms, 39-7 

linear prediction, 64-8 

positive definite matrices, 8-7, 46-13 
sparse matrices, 49-3 to 49-5 

sparse matrix factorizations, 49-3 
symmetric factorizations, 38-15 


Cholesky factorization with pivoting, 46-10 
Cholesky-like factorization, 8-9 

Chop, Mathematica software, 73-25 
Chordal bipartite graph 


bipartite graphs, 30-1 


Chordal distance 


eigenvalue problems, 15-10 


holeskyDecomposition, Mathematica software, 73-18, 
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Chordal graph 
bipartite graphs, 30-1 
Chordal graphs, 35-2 
Chordal symmetric Hamiltonian, minimally, 
35-15 
Chromatic index, 27-10 
Chromatic number, 28-9 
Chu, Ka Lok, 53-14 
Circulant matrices 
Maple software, 72-18 
nonnegative inverse eigenvalue problems, 20-5 
structured matrices, 48-2 
circul function, Matlab software, 71-6 
Circumcenter, 66-8 
Circumscribed hypersphere, 66-8 
Class functions, orthogonality relations, 68-6 
Classical Gram-Schmidt (CGS) scheme, 44-3 to 
44-4 
Classical groups, 67—5 to 67-7 
Classical Turing machine, 62-2 
Classification, states, 54—7 to 54-9 
Classifications I and II, 24~—7 to 24-11 
Classification theorem, 24—9, 24-11 
Clifford algebra, 69-13 
Clifford’s theorem 
characters, 68-9 
matrix groups, 67-2 
Cline, Alan Kaylor, 45-1 to 45-12 
Clique 
graph parameters, 28-9 
graphs, 35-2 
Clique number, 28-9 
Close cones, 8-10 
Closed ball of radius, 61-2 
Closed halfspace, 66—2 
Closed-loop ODE, 57-2 
Closed-loop system, 57-13 
Closed subset, 50-13 
Closed under matrix direct sums, 35-2 
Closed under permutation similarity, 35-2 
Closed under taking principal submatrices, 35-2 
Closed walk, 54—5 
Closure under addition, 1-2 
Closure under scalar multiplication, 1-2 
Cockades, 30-4 
Cocktail party graph, 28-4 
Coclique, 28-9 
Co-cover, 27-2 
Codewords, 61-1 
Codimension, 66-2 
Coding theory 
convolutional codes, 61—11 to 61-13 
distance bounds, 61-5 to 61-6 
fundamentals, 61-1 to 61-2 
linear block codes, 61-3 to 61-4 
linear code classes, 61-6 to 61-11 
main linear coding problem, 61-5 to 61-6 
Codomain, 3-1 
Coefficient matrices 
Bezout domains, 23-9 
linear differential equations, 55-1 
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Coefficients 

determination, 52-8 

multiple determination, 52-8 

systems of linear equations, 1-9 
Cofactors, determinants, 4—1 
Cogame utility, ATLAST, 71-20, 71-22 
Co-index, 9-7 
Colin de Verdiére parameter, 28-9, 28-10 
Collatz-Wielandt sets 
cone invariant departure, matrices, 26-3 to 
26-5 
irreducible matrices, 9-5 
Collineation, projective spaces, 65-7 
Colonius, Fritz, 56—1 to 56-21 
Color class, 28-9 
ColorFunction, Mathematica software, 73-27 
colormap command, Matlab software, 71-15 
colspace command, Matlab software, 71-17 
Columns 

equivalence, 23-5 

feasibility, 50-14 

indices, 23-9 

linear independence, span, and bases, 2-6 

matrices, 1-3 

pivoting, 46-5 

rank, 25-13 

signing, balanced, 33-5 

sign solvability, 33-5 

sum vector, 27—7 


vectors, 1-3 
Column-stochastic matrices, 9-15 
Combinatorial matrix theory 

algebraic connectivity, 36-1 to 36-11 

bipartite graphs and matrices, 30—1 to 30-10 

classes of (0,1)-matrices, 27—7 to 27-10 

completion problems, 35-1 to 35-20 

convex polytopes, 27-10 to 27-12 

digraphs, 29-1 to 29-13 

D-optimal matrices, 32-1 to 32-12 

doubly stochastic matrices, 27-10 to 27-12 

fundamentals, 27—1 to 27-12 

graphs, 28-1 to 28-12 

monotone class, 27—7 to 27-8 

multiplicity lists, 33-1 to 33-17 

permanents, 31—1 to 31-13 

sign-pattern matrices, 33-1 to 33-17 

square matrices, 27-3 to 27-6 

strong combinatorial invariants, 27-3 to 27-5 

structure and invariants, 27—1 to 27-3 

tournament matrices, 27—8 to 27-10 

weak combinatorial invariants, 27—5 to 27-6 
Combinatorial orthogonality, 33-16 
Combinatorial problems, 47-10 
Combinatorial symmetric partial matrix, 35-2 
Combinatorial symmetric sign pattern matrices, 

33-9 

Communication 

irreducible classes, 54-5 

irreducible matrices, 29-6 

nonnegative and stochastic matrices, 9-2 
Commutative algebras, 69-2 
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Commutative ring 
certain integral domains, 23-1 
permanents, 31-1 
Commutativity and Jordan identity, 69-12 
Commutativty axiom, 1-1 
Commutators 
nonassociative algebra, 69-2 
Schrédinger’s equation, 59-7 
Commute 
matrices, 1-4 
vector spaces, 3—2 
Companion matrices 
eigenvalues and eigenvectors, 4-6 
rational canonical forms, 6-8 
CompanionMatrix, Maple software, 72-17 
Comparison matrices, 19-9 
Compatibility, 33-2 
Complement 
binary matrices, 31-5 
fundamentals, P-1 
graphs, 28-2 
Complementary parameters, 64-8 
Complementary projections, 3-6 
Complementary slackness 
duality, 50-14 
duality and optimality conditions, 51-6 
max-plus eigenproblem, 25-7 
Complete bipartite graphs, 28—2, 30-1, see also Bipartite 
graphs 
Completed max-plus semiring, 25-1 
Complete flag, 56-9 
Complete graphs, 28-2, see also Graphs 
Completely positive matrices 
completion problems, 35-10 to 35-11 
nonnegative factorization, 9-22 
Complete min-plus semiring, 25-1 
Complete orthogonal decomposition, 39-11 
Complete orthogonal set, 5-3 
Complete pivoting, 38—10, see also Pivoting 
Complete reducibility 
matrix group, 67-1 
modules, 70-7 
square matrices, weak combinatorial invariants, 27-5 
Completion problems, matrices 
completely positive matrices, 35-10 to 35-11 
copositive matrices, 35-11 to 35-12 
doubly nonnegative matrices, 35-10 to 35-11 
entry sign symmetric P-, Po,,- and Po-matrices, 35-19 
to 35-20 
entry weakly sign symmetric P-, Pp,;- and 
Po-matrices, 35-19 to 35-20 
Euclidean distance matrices, 35—9 to 35-10 
fundamentals, 35-1 to 35-8 
inverse M-matrices, 35-14 to 35-15 
M- and Mo- matrices, 35-12 to 35-13 
nonnegative P-, Po,;- and Po-matrices, 35-17 to 35-18 
P-, Po,1- and Po-matrices, 35-15 to 35-17 
positive definite matrices, 35-8 to 35-9 
positive P-matrices, 35-17 to 35-18 
positive semidefinite matrices, 35-8 to 35-9 
strictly copositive matrices, 35-11 to 35-12 
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Completion property, 35-2 
Complex conjugate, 68-5 
Complex elements, Maple software, 72-7 
Complexity, convolutional codes, 61-12 
Complex numbers 
fundamentals, P—1 to P-2 
nonassociative algebra, 69-4 
Complex polynomial, 19-3 
Complex sign patterns, 33-14 to 33-15 
Component backward stable, 38-6 
Component-wise perturbation theory, 39-7 
Component-wise relative backward errors, linear 
system, 38-2 
Component-wise relative distance, 46-10 
Composite cycle, 33-2 
Composition, symmetric and Grassmann tensors, 
13-13 
Composition algebras, 69-8 to 69-10 
Compound matrix, 4-3 
Compressed column storage (CCS) scheme, 40-3 
Compressed row storage (CRS) scheme, 40—4 
Computational linear algebra 
fast matrix multiplication, 47-1 to 47-10 
large-scale matrix computations, 49-1 to 49-15 
structured matrix computations, 48-1 to 48-9 
Computational methods, 69-20 to 69-25 
Computational modes, 76-8 to 76-9 
Computational software 
freeware, 77-1 to 77-3 
Maple, 72-1 to 72-21 
Mathematica, 73-1 to 73-27 
Matlab, 71-1 to 71-22 
Computational software, subroutine packages 
ARPACK, 76-1 to 76-10 
BLAS, 74-1 to 74-7 
EIGS, 76-1 to 76-10 
LAPACK, 75-1 to 75-23 
Computations 
pseudospectra, 16-11 to 16-12 
singular value decomposition, 45—1 to 45-12 
Computations, large-scale matrices 
Arnoldi process, 49-10 to 49-11 
dimension reduction, 49-14 to 49-15 
eigenvalue computations, 49-12 
fundamentals, 49-1 to 49-2 
Krylov subspaces, 49-5 to 49-6 
linear dynamical systems, 49-14 to 49-15 
linear systems, equations, 49-12 to 49-14 
nonsymmetric Lanczos process, 49-8 to 49-10 
sparse matrix factorizations, 49-2 to 49-5 
symmetric Lanczos process, 49-6 to 49-7 
Computations, structured matrices 
direct Toeplitz solvers, 48-4 to 48-5 
fundamentals, 48—1 to 48-4 
iterative Toeplitz solvers, 48-5 
linear systems, 48-5 to 48-8 
total least squares problems, 48-8 to 48-9 
Computer science applications 
coding theory, 61-1 to 61-13 
information retrieval, 63—1 to 63-14 
quantum computation, 62-1 to 62-19 
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signal processing, 64—1 to 64-18 

Web searches, 63—1 to 63-14 
Concentration, metabolites, 60-10 
Conceptual PageRank, 63-11, see also PageRank 
cond, Mathematica software, 73-18 
Condensation digraph, 29-7 
Condensation method, 4—4 
Conditioning 

condition numbers, 37—7 to 37-9 

linear systems, 37-9 to 37-11 


Conditioning of eigenvalues, Maple software, 72-15 


ConditionNumber, Maple software, 72-9 
Condition numbers 
linear system perturbations, 38-2 
polar decomposition, 15-8 
sensitivity, 39-7 
Conductance, 28-9 
Conductivity, 66-13 
Cone invariant departure, matrices 
Collatz-Wielandt sets, 26—3 to 26-5 
core, 26—5 to 26-7 
distinguished eigenvalues, 26-3 to 26-5 
elementary analytic results, 26-12 to 26-13 
fundamentals, 26-1 
K-reducible matrices, 26-8 to 26-10 
linear equations over cones, 26-11 to 26-12 
peripheral spectrum, 26-5 to 26-7 
Perron-Frobenius theorem, 26-1 to 26-3 
Perron-Schaefer condition, 26—5 to 26-7 
spectral theory, 26-8 to 26-10 
splitting theorems, 26—13 to 26-14 
stability, 26-13 to 26-14 
Cones 
Perron-Frobenius theorem, 26-1 to 26-3 
positive definite matrices, 8-10 
programming, 51-2 
reducible matrices, 26-8 
Conformal and conformable partitions, 10-1 
Conformal partitions, 10-1 
Congruence 
bilinear forms, 12-2 


Hermitian and positive definite matrices, 8-5 to 8-6 


Hermitian forms, 12-8 
linear inequalities and projections, 25-10 
sesquilinear forms, 12-6 

Conjugacy, 56-5 

Conjugate character, 68-9 

Conjugate Gradient (CG) algorithm 
convergence rates, 41-14 to 41-15 
Krylov space methods, 41-4, 41-6 


Conjugate Gradient Squared (CGS) algorithm, 41-8 


Conjugate partition, P—2 
Conjugates, linear dynamical systems, 56-5 
ConjugateTranspose, Mathematica software 
fundamentals, 73-27 
matrix algebra, 73-9, 73-11 
singular values, 73-17, 73-18 
Connected properties and components 
digraphs, 29-2 
graphs, 28-1, 28-2 
Consecutive ones property, 30-4 


Consistency 
least squares solution, 39-1 
linear differential equations, 55-2 
linear statistical models, 52-8 
matrix norms, 37-4 
systems of linear equations, 1-9 
Consistent sign pattern matrices, 33-9 
Constant Boolean function 
Deutsch-Jozsa problem, 62-9 
Deutsch’s problem, 62-8 
Constant coefficients 
differential equations, 55-1 to 55-5 
linear differential equations, 55-1 
Constant term, 1-9 
Constant vector, 1-9 
Constituents, characters, 68—5 


ConstrainedMax, Mathematica software, 73-27 
ConstrainedMin, Mathematica software, 73-27 


Constraint length, 61-12 

Constraint qualification 
semidefinite programming, 51-7 
strong duality, 51-7 

Contact matrix, 60-9 

Containment gap, 44-9 

Content index, Web search, 63-9 
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Continuation character backslash, Maple software, 72-2 


Continuous dynamical systems, 56-5 
Continuous invariants, 24-11 


Continuous real-valued functions, 2-2 
contour command, Matlab software, 71-15 


Contour integration, 11-10 
Contraction matrices, 18—9 
Contractions, graphs, 28—4 
Contragredient, 68-3 


Control design, LTI systems, 57-13 to 57-17 
Controllability Hessenberg form, 57-9 


Controllability matrix, 57—7 
Controlled-NOT gate 

quantum computation, 62-4 

universal quantum gates, 62-8 
Controller, LTI systems, 57-13 
Controlling random vectors, 52-4 
Control system, 57-2 
Control theory 

analysis, LTI systems, 57-7 to 57-10 


control design, LTI systems, 57-13 to 57-17 
frequency-domain analysis, 57-5 to 57-6 


fundamentals, 57—1 to 57-5 


LTI systems, 57—7 to 57-10, 57-13 to 57-17 


matrix equations, 57-10 to 57-11 
state estimation, 57—11 to 57-13 
Control vector, 57—2 
Convergence 


implicitly restarted Arnoldi method, 44-9 to 44-10 


nonnegative and stochastic matrices, 9-2 


reducible matrices, 9-8, 9-11 

Toeplitz matrices, 16-6 
Convergence rates 

CG, 41-14 to 41-15 

GMRES, 41-15 to 41-16 

MINRES, 41-14 to 41-15 
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Convergent regular splitting, 9-17 
Convex hull, 66-2 
Convexity 

affine spaces, 65-2 

Euclidean point space, 66-2 

fundamentals, P—2 

phase 2 geometric interpretation, 50-13 

vector norms, 37-3 

vector seminorms, 37—4 
Convex linear combination, 50-13 
Convex polytopes, 27—10 to 27-12 
Convolution 

Fourier analysis, 58-3 

signal processing, 64—2 
Convolutional codes, 61-11 to 61-13 
Convolution identities 

discrete theory, 58-10 

Fourier analysis, 58—5 
Coordinate matrix, 60-2 
Coordinates 

change of basis, 2-10 to 2-12 

Euclidean point space, 66-1 

NMR protein structure determination, 60-2 
Coordinate vectors, 60-2 
Coordinatization theorem, 69-13 
Copositive matrices, 35-11 to 35-12 
Coppersmith and Winograd studies, 47-9 
Coprime elements, 23-2 
Core, 26-5 to 26-7 
Corless, Robert M., 72-1 to 72-21 
Corner minor, 21-7, see also Principal minors 
Corrected seminormal equations, 39-6 
Correlation, random vectors, 52-3 
Correlation coefficient, 52-3 
Correlation matrix 

positive definite matrices, 8-6 

random vectors, 52—4 
Correlations and variates, 53—7 to 53-8 
Cosine and sine, 11-11 
Cospectral graphs, 28-5 
costs, Mathematica software, 73-24 
Coupling time, 25-9 
Courant-Fischer inequalities, 14—4 
Courant-Fischer theorem 

eigenvalues, 14—4 

Hermitian matrices, 8-3 
Covariance, random vectors, 52—3 
Covariance matrix 

positive definite matrices, 8-9 

random vectors, 52-3 
Cover, combinatorial matrix theory, 27-2 
Cramer’s Rule, 4—3, 37-17 
Craven and Csordas studies, 21-11 to 21-12 
Critical digraphs, 25-6, 25-7, see also Digraphs 
Critical vertices, 25-6 
Cross, Mathematica software, 73-3, 73-5 
Cross correlation, 64—4 
Cross-covariance matrix, 52—4 
Cross-positives, 26—13 
CrossProduct, Maple software, 72-3 
CRS (compressed row storage) scheme, 40-4 
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Csordas, Craven and, studies, 21-11 to 21-12 
Cubics, Mathematica software, 73-14, 73-15 
Cui, Feng, 60-13 
Cumulative distribution function, 52-2 
Cuthill-McKee algorithm, 40-16 
Cut lattice, 30-2 
Cut space, 30-2 
Cut vertices, 36-3 
Cycle-clique, 35-14 
Cycle conditions, 35-9 
Cycle of length, 28-1, 28-2 
Cycle products, digraphs, 29-4 to 29-6 
Cycles 
digraphs, 29-2 
Jacobi method, 42-18 
matrices, 48-2 
pattern, 33-9 
simplex method, 50-12 
time, 25-8 
Cycles of length, 33-2 
Cyclically real ray pattern, 33-14 
Cyclic code, 61-6 
Cyclicity, matrix power asymptotics, 25-8 
Cyclicity theorem, 25-8 
Cyclic normal form 
digraphs, 29-9 to 29-11 
imprimitive matrices, 29-10 
Cyclic simplexes, 66-12 


D 


Damped least squares, 39-9 to 39-10 

Dangling node, 63-11 

Daniel studies, 44—4 

Data Encryption Standard (DES) cryptography system, 

62-17 

Data fitting, 39-3 to 39-4 

Data matrix, 53-2 to 53-3 

Data perturbations, 38-2 

Datta, Biswa Nath, 37-1 to 37-21 

Davidson’s method, 43-10 

Day, Jane, 1-1 to 1-15 

DCT, see Discrete Fourier transform (DFT) 

DeAlba, Luz M., 4-1 to 4-11 

Decoder, 61-2 

Decoding, 61-2 

Decomposable tensors, 13-3 

Decomposition 
direct solution of linear systems, 38—7 to 38-15 
high relative accuracy, 46—7 to 46-10 
least squares solutions, 39-11 to 39-12 
Mathematica software, 73-18 to 73-19 
matrix group, 67-1 
Morse, dynamical systems, 56—7 to 56-9 
rank revealing decomposition, 39-11 to 39-12 
semisimple and simple algebras, 70-4 
singular values, 17-15 
symmetric and Grassmann tensors, 13-13 
symmetric factorizations, 38-15 
tensors, multilinear algebra, 13-7 
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Deconvolution 
Fourier analysis, 58-8 
functional and discrete theories, 58-16 
Decoupling Principle, 59-1, 59-2 
Deeper properties, 21-9 to 21-12 
Defective matrices, 4-6 
Definite matrices, see Positive definite matrices (PSD) 
Definite pencils, 15-10 
Deflation, 42-2 
Degenerate characteristics 
sesquilinear forms, 12-6 
simplex method, 50-12 
Degree 
certain integral domains, 23-2 
characters, 68—5 
control theory, 57-2 
convolutional codes, 61-11 
frequency-domain analysis, 57—6 
general properties, 69-5 
graphs, 28-2 
group representations, 68-1 
matrix group, 67-1 
matrix representations, 68-3 
max-plus permanent, 25-9 
Deletion, edges and vertices, 28—4 
Delsarte’s Linear Programming Bound, 28-12 
Delta function, 58-2 
demand, Mathematica software, 73-24 
Demmel, James, 75-1 to 75-23 
Demmel-Kahan singular value decomposition, 45-7 to 
45-8 
Denardo algorithm, 25-8 
Denman-Beavers iteration, 11-11 
Dense matrices 
fundamentals, 43-1 
large-scale matrix computations, 49-2 
software, 77—2 
techniques, 43-3 to 43-9 
de Oliveria studies, 20-3 
Depth, Jordan canonical form, 6-3 
Derangements, 31-6 
De Rijk’s row-cyclic pivoting, 46-4 
Derivation, Lie algebras, 70-1 
Derived algebra, 70-3 
Derogatory matrices, 4—6 
Desargues’ theorem, 65-8, 65-9 
DES (Data Encryption Standard) cryptography system, 
62-17 
Design, square case, 32-2 
Design matrices 
D-optimal matrices, 32-1 
linear statistical models, 52-8 
Det, Mathematica software, 73-10, 73-11 
det command, Matlab software, 71-17 
Detection, control theory, 57—2 
Determinant, Maple software, 72-5 
Determinantal region, 33-14 
Determinantal relations, 14—10 to 14-12 
Determinants 
advanced results, 4—3 to 4-6 
connections, 31-12 to 31-13 


fast matrix multiplication, 47-10 
fundamentals, 4—1 to 4-3 
invariants, 23-5 
Deterministic Markov decision process, 25-3 
Deterministic spectral estimation, 64-14 
det function, Matlab software, 71-3 
Deutsch-Jozsa problem, 62-9 to 62-11 
Deutsch’s problem, 62-8 to 62-9 
Developer’ HessenbergDecomposition,, Mathematica 
software, 73-27 
DGEMM BLAS subroutine package, 42-21 
DGKS mechanism, 44—4 
Dhillon, Inderjit S., 45-1 to 45-12 
diag, Mathematica software, 73-16 
Diagonal entry, 1-4, 23-5 
Diagonalization 
eigenvalue problems, 15-10 
eigenvalues and eigenvectors, 4-6, 4—7 to 4-8 
Diagonally dominant matrices, 9-17 
Diagonally scaled representation, 46-10 
Diagonally scaled totally unimodular (DSTU), 46-8, 
46-9 
Diagonal matrices, 1—4 
DiagonalMatrix, Mathematica software 
eigenvalues, 73-15, 73-16 
matrices, 73-6, 73-8 
Diagonal pattern, 33-2 
Diagonal product, 27-10 
Diagonals, square matrices, 27-3 
Diagonal stability, 19-9 
Diameter, eigenvalues, 34—7 
Diao, Zijian, 62—1 to 62-19 
Dias da Silva, José A., 13-1 to 13-26 
Differentiable functions, 56-5 
Differential-algebraic equations, 55-1 
Differential-algebraic equations of order, 55-2 
Differential equations 
constant coefficients, 55-1 to 55-5 
eigenvalues and eigenvectors, 4-10 to 4-11 
linear different equations, 55-1 to 55-5 
linear differential-algebraic equations, 55—7 to 55-10, 
55-14 to 55-16 
linearization, 56-19 
linear ordinary differential equations, 55-5 to 55-6, 
55-10 to 55-14 
stability, 55-10 to 55-16 
Differential quotient-difference (dqds) step, 45-8 to 
45-9 
Digits, Maple software, 72-14 
Digraphs 
adjacency matrix, 29-3 to 29-4 
cycle products, 29-4 to 29-6 
cyclic normal form, 29-9 to 29-11 
directed graphs, 29-3 to 29-4 
fundamentals, 29-1 to 29-3 
irreducible, imprimitive matrices, 29-9 to 29-11 
irreducible matrices, 29-6 to 29-8 
matrices, 29-3 to 29-4 
matrix completion problems, 35-2 
max-plus algebra, 25-2 
minimal connection, 29-12 to 29-13 
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modeling and analyzing fill, 40-11 

nearly reducible matrices, 29-12 to 29-13 
nonnegative and stochastic matrices, 9-2 
P-, Po,1- and Po-matrices, 35-15 to 35-16 


primitive digraphs and matrices, 29-8 to 29-9 


sign-pattern matrices, 33-2 
strongly connected digraphs, 29-6 to 29-8 
walk products, 29-4 to 29-5 
Dihedral interior angle, 66—7 
Dilations, numerical range, 18-9 to 18-10 
Dimension 
doubly stochastic matrices, 27-10 
Euclidean point space, 66-2 
Euclidean simplexes, 66-7 
grading, 70-9 
nonassociative algebra, 69-1 
simultaneous similarity, 24-8 
vector space, 2—3 
Dimensional projective spaces and subspaces, 
65-6 
Dimension reduction, 49-14 to 49-15 
Dimensions, Mathematica software, 73-6, 
73-7, 73-27 
Dimension theorem 
kernel, 3-5 
matrix range, 2-6 to 2-9 
null space, 2-6 to 2-9 
range, 3-5 
rank, 2-6 to 2-9 
Dirac’s bra-ket notation, 62—2 
Directed arcs, digraphs, 29-1 
Directed bigraphs, 30-4 
Directed digraphs, 29-1 
Directed edges, 29-1 
Directed graphs, 29-3 to 29-4 
Directed multigraph, 29-2 
Direction, arrival estimation, 64—15 to 64-18 
Direction space, affine spaces, 65-2 
Direct isometry, 65-5, see also Isometry 
Directory structure and contents, 76-3 
Direct solution, linear systems 
fundamentals, 38-1 
Gauss elimination, 38—7 to 38-12 
LU decomposition, 38-7 to 38-12 
orthogonalization, 38-13 to 38-15 
perturbations, 38-2 to 38-5 
QR decomposition, 38-13 to 38-15 
symmetric factorizations, 38-15 to 38-17 
triangular linear systems, 38-5 to 38-7 
Direct sum 
block diagonal and triangular matrices, 10-4 
decompositions, 2—4 to 2-6 
direct sum decompositions, 2-5 
group representations, 68-3 
nonassociative algebra, 69-3 
semisimple and simple algebras, 70-3 
Direct Toeplitz solvers, 48—4 to 48-5 
Dirichlet conditions, 59-10 
Discrete approximation, 58-13 
Discrete event systems, 25-3 to 25-4 
Discrete Fourier transform (DFT), 58-9 
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Discrete invariants, 24-11 
Discrete stochastic process, 54—1 
Discrete theory, 58—2 to 58-17 
Discrete time Fourier transform, 64—2 
Discrete variables, 52-2 
Discrete Wiener filtering problem, 64-10 
Discriminant coordinates, 53-6 
Disjoint matrix multiplication, 47-8 
Dispersion, 21-7 
Dispersion matrix, 52-3 
Dissection strategy, 40-16 
Dissimilarity, 53-13 
Dissimilarity matrix, 53-14 
Distance 
coding theory, 61-2 
Euclidean point space, 66-2 
Euclidean spaces, 65—4 
graphs, 28-1 
inner product spaces, 5-1 
NMR protein structure determination, 60-2 
numerical range, 18-1 
Distance bounds, 61—5 to 61-6 
Distance matrix, 60-2 
Distance-regular graph, 28-11 
Distinguished eigenvalues 


cone invariant departure, matrices, 26-3 to 26-5 


nonnegative and stochastic matrices, 9-2 
Distinguished face, 26-5 
Distributivity, 69-1 
Distributivity axiom, 1-1 
Divide and Conquer Bidiagonal singular value 

decomposition, 45-10 

Divide and conquer method, 42-12 to 42-14 
Division algebra, 69-2, 69-4 
Division Property, 6-11 
DLARGV LAPACK subroutine, 42-9 
DLARTV LAPACK subroutine, 42-9 
DLAR2V LAPACK subroutine, 42-9 
Documents, retrieved, 63-2 
Document vector, 63-1 
Domain, linear transformations, 3—1 
Domain decomposition methods, 41-12 
Dominant eigenvalue, 42-2 
Dong, Qunfeng, 60-13 


Dongarra, Jack, 74-1 to 74-7, 75-1 to 75-23, 77-1 to 


77-3 
D-optimal matrices 
balanced matrices, 32-12 
fundamentals, 32-1 
nonregular matrices, 32—7 to 32-9 
nonsquare case, 32—2 to 32-12 
regular matrices, 32—5 to 32-7 
square case, 32—2 to 32-4 
DORGTR LAPACK subroutine, 42-8 
Dot, Mathematica software 
matrices, 73-8 
vectors, 73-4 
Dot product 
Bernstein-Vazirani problem, 62-11 
floating point numbers, 37-14 
inner product spaces, 5-2 
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DotProduct, Maple software, 72-3 
Double directed tree, 29-2 
Double echelon form, 21—2 
Double generalized stars, 34-11 to 34-14 
Double path, generalized stars, 34-11 
Double precision, 37-13 
Double stars, 34—11 
Doubly nonnegative matrices, 35-10 to 35-11 
Doubly stochastic matrices 
combinatorial matrix theory, 27-10 to 27-12 
fundamentals, 9-15 
permanents, 31-3 to 31-4 
Downdating, 39-8 to 39-9 
Downer branch, 34—2 
Downer vertex, 34—2 
Drazin inverse, 55-7 
Drmac, Zlatko, 46-1 to 46-16 
Drop, Mathematica software 
fundamentals, 73-27 
matrices manipulation, 73-13 
vectors, 73-3 
Drop-off condition, 21-12 
DROT BLAS routine, 42-9 
DSBTRD LAPACK subroutine, 42-9 
DSDRV ARPACK routine, 42-21 
D-stability, 19-5 to 19-7 
DSTEBZ LAPACK subroutine, 42-15 
DSTEDC LAPACK subroutine, 42-14 
DSTEGR LAPACK subroutine, 42-17 
DSTEIN LAPACK subroutine, 42—15 
DSTU (diagonally scaled totally unimodular) matrices 
rank revealing decomposition, 46-8 
rank revealing decompositions, 46-9 
DSYEV LAPACK subroutine, 42-11 
DSYTRD LAPACK subroutine, 42-8 
Duality 
code, 61-3 
cones, 8-11, 26—2 
control theory, 57-2 
inner product spaces, 13-23 
linear functionals and annihilator, 3-8 


linear programming, 50-13 to 50-17 
optimality conditions, 51-6 
projective spaces, 65-7 
rectangular matrix multiplication, 47-5 
semidefinite programming, 51-5 to 51-7 
vector norms, 37—2 
Duality theorem, 51-7 
Dual linear program, 25-5 
Dulmage-Mendelsohn Decomposition theorem, 
27-4 
Dynamical systems 
chain recurrence, 56—7 to 56-9 
eigenvalues and eigenvectors, 4-11 
flag manifolds, 56-9 to 56-11 
Floquet theory, 56-12 to 56-14 
fundamentals, 56—1 to 56-2 
Grassmannians, 56-9 to 56-11 
linear differential equations, 56—2 to 56-4 
linear dynamical systems, 56-5 to 56—7 
linearization, 56-19 to 56-21 


linear skew product flows, 56-11 to 56-12 
Morse decompositions, 56—7 to 56-9 
periodic linear differential equations, 56-12 to 
56-14 
random linear dynamical systems, 56-14 to 56-16 
robust linear systems, 56-16 to 56-19 
Dynamic compensator, 57-13 
Dynamic programming, 25-3 


E 


Eckart-Young low rank approximation theorem, 5-11 
EDD (elementary divisor domain), 23-2 
ED (Euclidean domain), 23-2 
Edge cut, 36-3 
Edges 
digraphs, 29-1 
Euclidean simplexes, 66—7 
graphs, 28-1, 28-4 
ED-RCF (elementary divisors rational canonical form) 
matrix, 6-8 to 6-9 
Effects of reordering, 40-14 to 40-18 
Efficacy, methods comparison, 42-21 
Efficiency, error analysis, 37-16 to 37-17 
egn, Mathematica software, 73-21 
egns, Mathematica software, 73-20 
eig command, Matlab software 
eigenvalues, 15-5 
fundamentals, 71-9, 71-17, 71-18 
implicitly restarted Arnoldi method, 44-1 
Lanczos methods, 42—21 
EigenConditionNumbers, Maple software, 72-15 
Eigenpairs 
eigenvalue problems, 15-10 
matrix perturbation theory, 15-1 
Eigenproblems 
max-plus algebra, 25-6 to 25-8 
nonsymmetric, LAPACK subroutine package, 75-17 
to 75-20 
Eigensharp characteristic, 30-8 
Eigenspaces 
eigenvalues and eigenvectors, 4-6 
max-plus eigenproblem, 25-6 
nonnegative and stochastic matrices, 9-2 
Eigensystem, Mathematica software, 73-14, 73-15, 
73-16 
Eigentriplets 
eigenvalue problems, 15-9 to 15-10 
matrix perturbation theory, 15-1 
Eigenvalue decomposition (EVD), 42-2 
Eigenvalues, high relative accuracy 
accurate SVD, 46-2 to 46-5, 46—7 to 46-10 
fundamentals, 46-1 to 46-2 
one-sided Jacobi SVD algorithm, 46-2 to 46-5 
positive definite matrices, 46-10 to 46-14 
preconditioned Jacobi SVD algorithm, 46-5 to 46-7 
rank revealing decomposition, 46—7 to 46-10 
structured matrices, 46-7 to 46-10 
symmetric indefinite matrices, 46-14 to 46-16 
Eigenvalues, Maple software, 72-11, 72-12, 72-14 
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Eigenvalues, Maple software, 72-15 
Eigenvalues, Mathematica software 
eigenvalues, 73-14, 73-15 
fundamentals, 73-27 
singular values, 73-17 
Eigenvalues, numerical methods 
high relative accuracy computation, 46-1 to 46-16 
implicitly restarted Arnoldi method, 44-1 to 44-12 
iterative solution methods, 41-1 to 41-17 
singular value decomposition, 45-1 to 45-12 
symmetric matrix techniques, 42—1 to 42-22 
unsymmetric matrix techniques, 43-1 to 43-11 
Eigenvalues, problems 
generalized, 15-9 to 15-11 
perturbation theory, 15-1 to 15-6, 15-9 to 15-11 
relative perturbation theory, 15-13 to 15-15 
Eigenvalues, symmetric matrix techniques 
bisection method, 42—14 to 42-15 
comparison of methods, 42-21 to 42-22 
divide and conquer method, 42-12 to 42-14 
fundamentals, 42-1 to 42-2 
implicitly shifted QR method, 42-9 to 42-11 
inverse iteration, 42-14 to 42-15 
Jacobi method, 42-17 to 42-19 
Lanczos method, 42-19 to 42-21 
method comparison, 42-21 to 42-22 
methods, 42—2 to 42-5 
multiple relatively robust representations, 42-15 to 
42-17 
tridiagonalization, 42-5 to 42-9 
Eigenvalues, unsymmetric matrix techniques 
dense matrix techniques, 43-3 to 43-9 
fundamentals, 43-1 
generalized eigenvalue problem, 43-1 to 43-3 
sparse matrix techniques, 43-9 to 43-11 
Eigenvalues and eigenvectors 
adjacency matrix, 28-5 
canonical forms, 6—2 to 6-3 
cone invariant departure, matrices, 26-3 to 26-5 
fundamentals, 4—6 to 4-11 
generalized eigenvalue problem, 43-1 to 43-3 
graphs, 28-5 to 28-7 
Hermitian matrices, 17-13 to 17-14 
inequalities, 17-9 to 17-10 
LAPACK subroutine package, 75-9 to 75-11, 75-11 to 
75-13 
large-scale matrix computations, 49-12 
Maple software, 72-11 to 72-12 
Mathematica software, 73-14 to 73-16 
Matlab software, 71-9 
matrix equalities and inequalities, 14-1 to 14-5, 14-8 
to 14-10 
multiplicity lists, 34—7 to 34-8 
numerical stability and instability, 37-20 
pseudoeigenvalues and pseudoeigenvectors, 16-1 
reducible matrices, 9-8 to 9-9, 9-11 
Schrédinger’s equation, 59-7 
sign-pattern matrices, 33-9 to 33-11 
singular values and singular value inequalities, 17-13 
to 17-14 
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sparse eigenvalue solvers, software, 77-2 
spectrum and boundary points, 18-3 
Eigenvectors, Maple software, 72-11 
Eigenvectors, Mathematica software, 73-14 
eigshow command, Matlab software, 71-9 
EIGS subroutine package, 76-9 to 76-10 
eigtool, Matlab software, 16-12, 71-20 
Eijkhout, Victor, 74-1 to 74-7, 77-1 to 77-3 
Electron density distribution function, 60-7 
Electrostatics, 59-11 
Element, Mathematica software, 73-24 
Elementary analytic results, 26-12 to 26-13 
Elementary bidiagonal matrices, 21-5 
Elementary column operations, 6-11 
Elementary divisor domain (EDD), 23-2 
Elementary divisors, 6-8 to 6-11 
Elementary divisors rational canonical form (ED-RCF) 
matrix, 6-8 to 6-9 
Elementary matrices, 1-12 
Elementary row operations 
Gaussian and Gauss-Jordan elimination, 1—7 
matrix equivalence, 23-5 
Smith normal form, 6-11 
Elementary symmetric function, P-2 to P-3 
Element of volume, 13—25 
Eliminate, Mathematica software, 73—20, 73-23 
Elimination graph, 40-11 
Elimination ordering, 40-14 
Elimination sequence, 40-14 
ELMAT directory, Matlab software, 71-5 
Elsner theorem, 15-2 
Embedding graphs, 28-3 
Embree, Mark, 16-1 to 16-15 
Empty graphs, 28-2 
Empty matrix, 68-3 
Encoder 
coding theory, 61-1 
linear block codes, 61-3 
Encoding, 61-2 
Endpoint, 28-1 
Energy norm, 37-2 
Entry, matrices, 1-3 
Entry sign symmetric P-, Po,,- and Po-matrices, 35-19 
to 35-20 
Entry weakly sign symmetric P-, Po,;- and Po-matrices, 
35-19 to 35-20 
Envelope, reordering effect, 40-14 
Envelope method, 40-16 
Epsilon-pseudospectrum, square matrix, 71-20 
eqns, Mathematica software, 73-20 
Equalities and inequalities, matrices 
determinantal relations, 14—10 to 14-12 
eigenvalues, 14—1 to 14-5, 14-8 to 14-10 
inversions, 14—15 to 14-17 
nullity, 14-12 to 14-15 
rank, 14-12 to 14-15 
singular values, 14-8 to 14-10 
spectrum localization, 14-5 to 14-8 
Equality-constrained least squares problems, 75-6 to 
75-7 
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Equal matrices, 1-3 
Equations of motion, 59-3 
Equation solving, Maple software, 72-9 to 72-11 
Equicorrelation matrix and structure, 52-9 
Equilibrium 
linear dynamical systems, 56-5 
matrix games, 50-18 
Equivalence 
bilinear forms, 12-2 
group representations, 68-1 
Hermitian forms, 12-8 
linear block codes, 61-3 
linear dynamical systems, 56-5 
linear independence, span, and bases, 2—7 
matrix equivalence, 23-5 
matrix representations, 68-3 
scaling nonnegative matrices, 9-20 
sesquilinear forms, 12-6 
systems of linear equations, 1-9 
trees, 34-8 
vector norms, 37-3 
Equivalence class modulo, 25-10 
Equivalence relation, P-3 
Ergodic class matrices, 9-15 
Ergodic flow, 56-14 
Ergodicity coefficient 
bounds, 9-5 
irreducible matrices, 9—5 
nonnegative and stochastic matrices, 9-2 
reducible matrices, 9-10 to 9-11 
Error analysis 
algorithms, 37-16 to 37-17 
conditioning and condition numbers, 37—7 to 37-9 
efficiency, 37-16 to 37-17 
floating point numbers, 37-11 to 37-16 
fundamentals, 37-1 to 37-2 
linear systems conditioning, 37-9 to 37-11 
matrix norms, 37—4 to 37-6 
numerical stability and instability, 37-18 to 37-21 
vector norms, 37—2 to 37-3 
vector seminorms, 37-3 to 37—4 
Errors 
estimation, 41-16 to 41-17 
Krylov subspaces and preconditioners, 41-2 
linear prediction, 64-7 
LTI systems, 57-14 
protection, coding theory, 61-1 
vectors, 61-2 
ESPRIT algorithm, 64-17 
Estimated principal components, 53-5 
Estimation, correlations and variates, 53-7 to 53-8 
Estimation, least squares, 53-11 to 53-12 
Estimation error, 57-14 
Estimation of state, 57-11 to 57-13 
Euclidean algorithm, 6-11 
Euclidean distance 
Euclidean point space, 66—2 
metric multidimensional scaling, 53-14 
Euclidean distance matrices, 35—9 to 35-10 
Euclidean Distance Matrix, 51-9 


Euclidean domain (ED) 
certain integral domains, 23-2 
matrix equivalence, 23-6 
matrix similarity, 24-2 
Euclidean geometry 
fundamentals, 66-1 
Gram matrices, 66—5 to 66—7 
point spaces, 66-1 to 66-5 
resistive electrical networks, 66-13 to 66-15 
simplexes, 66-7 to 66-13 
Euclidean norm 
matrix norms, 37-4 
one-sided Jacobi SVD algorithm, 46-4 
preconditioned Jacobi SVD algorithm, 46-6, 
46-7 
rank revealing decompositions, 46-8 
symmetric indefinite matrices, 46-14 
vector norms, 37—2 
Euclidean Parallel Postulate, Generalized, 65-3 
Euclidean properties 
matrices, 53-14 
plane, 65-4 
point space, 66-1 to 66-5 
projective spaces, 65-9 
simplexes, 66-7 to 66-10 
unitary similarity, 7-2 
Euclidean spaces 
Gram matrices, 66-5 
inner product spaces, 5-2 
orthogonality, 5-5 
semisimple and simple algebras, 70-4 
vector spaces, 1-3 
Euclid’s algorithm 
certain integral domains, 23-2 
matrix equivalence, 23-7 
Evaluation, permanents, 31-11 to 31-12 
EVD (eigenvalue decomposition), 42-2 
Even cycle, 33-2 
Exact breakdown, 49-8 
Exactly universal quantum gates, 62-7 
Exact numerical cancellation, 40—4 
Exact shifts, 44-6 
Exceptional, Jordan algebra 
Jordan algebras, 69-12 
nonassociative algebra, 69-3 
Exogenous input, 57-14 
Expand, Mathematica software, 73-25 
Expanders, graph parameters, 28-9 
Expansion 
Bezout domains, 23-9 
determinants, 4—1 to 4—2 
reducible matrices, 9-8, 9-11 to 9-12 
exp command, Matlab software, 71-19 
Expectation, 52-2, 52-3 
Expected value 
random vectors, 52-3 
Schrédinger’s equation, 59-7 
state estimation, 57-12 
statistics and random variables, 52-2 
Explicit QR iteration, 43—4 to 43-5 
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Explicit restarting, 44-6 
expm function, Matlab software, 11-11 
Exponentials, 56-2 
Exponential stability 
linear differential-algebraic equations, 55-14 
linear differential equations, 56-3 
linear ordinary differential equations, 55-10 
Exponent of matrix multiplication complexity, 47-2 
Exponents 
floating point numbers, 37-11 
linear skew product flows, 56-11 
primitive digraphs and matrices, 29-9 
expr, Mathematica software, 73-26 
Extended colored graphs, 66-10 
Extended graphs, 66-10 
Extended precision, 37-13 
Extended Tam-Schneider condition, 26-7 
Extendible characters, 68—9 
Extensions 
pseudospectra, 16-12 to 16-15 
sign solvability, 33-5 
Exterior algebra, 70-2 
Exterior point method, 50-23 
Exterior power, 13-12 
Exterior product, 13-13 
External direct sum 
direct sum decompositions, 2-5 
nonassociative algebra, 69-3 
External product, 13-25 
Extremal generators, 25-12 
Extreme pathways, 60-10 
Extreme point, 50-13 
Extreme ray 
flux balancing equation, 60-10 
linear independence and rank, 25-12 
Extreme values, 27-7 
Extreme vectors, 26-2 
ezcontour command, Matlab software, 71-15 
ezplot command, Matlab software, 71-14, 71-14 to 
71-15, 71-17 
ezsurf command, Matlab software, 71-17 


F 


Faber and Manteuffel theorem, 41-8 
Face 
Collatz-Wielandt sets, 26—5 
Euclidean simplexes, 66—7 
geometry, 51-5 
Perron-Frobenius theorem, 26-2 


reducible matrices, 26-8 
Facially exposed face, 51-5 
Factorizations, see also QR factorization 
Arnoldi factorization, 44—2 to 44-4 
bipartite graphs, 30-8 to 30-10 
function computation methods, 11-9 to 11-10 
information retrieval, 63—5 to 63-8 
LU factorizations, 1-13 
nonnegative matrix factorization, 63—5 to 63-8 
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orthogonal, least squares solutions, 39-5 to 39-6 
sparse matrix methods, 40—4 to 40-10 
total positive and total negative matrices, 21—5 to 21-6 


Factorizations, direct solution of linear solutions 


fundamentals, 38—1 

Gauss elimination, 38—7 to 38-12 

LU decomposition, 38-7 to 38-12 
orthogonalization, 38-13 to 38-15 
perturbations, 38-2 to 38-5 

QR decomposition, 38-13 to 38-15 
symmetric factorizations, 38-15 to 38-17 
triangular linear systems, 38—5 to 38-7 


Faithful characteristics 


characters, 68—5 
group representations, 68-2 
matrix representations, 68-3 


Fallat, Shaun M., 21-1 to 21-12 

Fast algorithms, 47-2 to 47-4, see also Algorithms 
Fast Fourier transform (FFT), 58-17 to 58-20, 60-8 
Fast large-scale matrix computations, 49-2 

Fast matrix inversion, 47—9 to 47-10 

Fast matrix multiplication 


advanced techniques, 47—7 to 47-9 
algorithms, 47-2 to 47-5 

applications, 47-9 to 47-10 
approximation algorithms, 47-6 to 47-7 
fundamentals, 47—1 to 47-2 


F-distribution, 53-10 

Feasible region, 50-1 

Feasible solutions, 50-1 

Feasible values, 50-1 

Feedback controller, 57-13 

Fejer’s theorem, 8-10 

Fekete’s Criterion, 21—7 

Fermions, 59-10 

FFT, see Fast Fourier transform (FFT) 
Fiedler, Miroslav, 66—1 to 66-15 
Fiedler vectors, 36—1, 36—4, 36—7 to 36-9 
Field of relational functions, 23-2 
Field of values 


convergence rates, 41-16 
numerical range, 18-1 


Fields, P-3 

Fill element, 40-4 

Fill graphs, 40-14 

Fill-in, sparse matrix factorizations, 49-3 
Fill matrix, 40-4 

Filter polynomial, 44-6 

Final class, 54-5 

Final subset, 9-2 

Fine, Morse decomposition, 56—7 

Finite dimensional 


direct sum decompositions, 2-5 
nonassociative algebra, 69-2 
vector space, 2—3 


Finite energy, 64—2 
Finite impulse response (FIR) 


adaptive filtering, 64-12 
signal processing, 64-2, 64-3 
Wiener filter, 64-11 
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Finitely generated elements 
Bezout domains, 23-8 
max-plus algebra, 25-2 
Finitely generated ideals, 23-2 
Finite Markov chain, 54—2 
Finite Markov chains, 54—9 to 54-11 
Finite power, 64—5 
Finite precision arithmetic, 41-16 to 41-17 
Finite time exponential growth rate, 56-16 
FIR (finite impulse response) 
adaptive filtering, 64-12 
signal processing, 64—2, 64-3 
Wiener filter, 64-11 
First, Mathematica software 
matrices manipulation, 73-13 
singular values, 73-18 
vectors, 73-3 
First level radix 2 FFT, 58-17 to 58-18 
First (population) canonical correlations and 
variates, 53-7 
FIR Wiener filtering problem, 64-10 
Fischer’s Determinantal Inequality, 8-10 
Fischer’s inequality, 14-11 
Fixed point 
linear dynamical systems, 56-5 
linearization, 56-19 
Fixed spaces, 3-6 
Flag manifolds, 56-7, 56-9 to 56-11 
Flatten, Mathematica software 
fundamentals, 73-27 
linear programming, 73-24 
matrices manipulation, 73-14 
Flexible algebra, 69-10 
Flip map, 22-2 
Floating point numbers, 37-11 to 37-16 
Floating point operation (flop) 
algorithms and efficiency, 37-16 
large-scale matrix computations, 49-2 
Floquet exponents, 56-17 
Floquet theory 
dynamical systems, 56-12 to 56-14 
random linear dynamical systems, 56-15 to 
56-16 
Flow lattice, 30-2 
Flux balancing equation, 60-10 to 60-13 
fname command, Matlab software, 71-10, 71-11 
FOM (Full Orthogonalization Method), 41-7 
Ford-Fulkerson theorem, 27—7 
Forest, graphs, 28-2 
for loops, Matlab software, 71-11 
format short command, Matlab software, 71-8 
Formed space, 67-5 
Formulating linear programs, 50-3 to 50-7 
Formulation, 50-3 to 50-7 
Forward errors, 37-18, 37—20 to 37-21 
Forward stability, 37-18 
Four (4)-cockades, 30-4 
Fourier analysis 
discrete theory, 58-8 to 58-17 
fast Fourier transform, 58-17 to 58—20 
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function/functional theory, 58-2 to 58-8, 58-12 to 
58-17 
fundamentals, 58-1 
Fourier coefficients, 5—4 
Fourier expansion, 5-4 
Fourier transforms 
Green’s functions, 59-10 
Karle-Hauptman matrix, 60-8 
Frame, 56—9 
Frameticks, Mathematica software, 73-27 
F-ratio, 53-3 
Free algebras, 69-18 
Free distance, 61—12 
Free Lie algebra, 70-2 
Free variables, 1-10 
Freeware (software), 77—1 to 77-3 
Frequency-domain analysis, 57-5 to 57-6 
Frequency response, 64—2 
Freund, Roland W., 49-1 to 49-15 
Friedland, Shmuel, 23-1 to 23-10, 24—1 to 24-11 
Friendship theorem, 28-7 
Frobenius inequality, 14-13 
Frobenius-Kénigh theorem, 27-4 
Frobenius norm 
eigenvalues, 15-4 
elementary analytic results, 26-12 
irreducible matrices, 29-7 
matrices function behavior, 16-10 
matrix norms, 37—4 
protein structure comparison, 60-4 
semidefinite programming, 51-3 
singular value decomposition, 5-11 
square matrices, 27-6, 29-11 
unitarily invariant norms, 17-6 
unitary similarity, 7-2 
weak combinatorial invariants, 27-6 
Frobenius normal form, 27-5 
Frobenius reciprocity, 68-9, 68-10 
Frobenius- Victory theorem, 26-8, 26-9 
Frontal/multifrontal methods, 40-10 
Front end, Mathematica software, 73-1 
Frucht-Kantorovich Inequality, 52-10 
Full cones, 8-10 
Full level radix 2 FFT, 58-18 to 58-19 
Full Orthogonalization Method (FOM), 41-7 
Full rank least squares problem, 5-14 
Full-rank model, 52-8 
Full reorthogonalization procedure, 42-20 to 42-21 
Fully indecomposable, 27-3 to 27-4 
Fulton studies, 17-13 
Functional inequalities 
irreducible matrices, 9-5 
reducible matrices, 9-10 
Function evaluation operator, 62-5 to 62-6 
Function/functional theory, 58-2 to 58-8, 58-12 to 
58-17 
Functions, Matlab software, 71-11 
Functions of matrices 
computational methods, 11-9 to 11-12 
cosine, 11-7 to 11-8 
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exponential, 11-5 to 11-6 

fundamentals, 11-1 

logarithm, 11-6 to 11-7 

sign function, 11-8 

sine, 11-7 to 11-8 

square root, 11-4 to 11-5 

theory, 11-1 to 11-4 
Fundamental period, 58-2 
Fundamental subspaces, 39-4 
Fundamental tensor, 13-25 


G 


Galerkin condition 
Arnoldi factorization, 44—3 
Krylov subspace projection, 44-2 
Gale-Ryser theorem, 27-7 
gallery command, Matlab software, 71-5 
Gantmacher-Lyapunov theorem, 26-14 
Gantmacher studies, 19—4 
Gaubert, Stéphane, 25-1 to 25-14 
Gaussian elimination, see also LU factorizations 
algorithm efficiency, 37-17 
bipartite graphs, 30-7 
bisection and inverse iteration, 42-15 
direct solution of linear systems, 38-7 to 38-12 
fundamentals, 1—7 to 1-9 
Karle-Hauptman matrix, 60-8 
modeling and analyzing fill, 40-10 
numerical stability and instability, 37-20, 37-21 
reordering effect, 40-16, 40-18 
sparse matrix factorizations, 40-5, 40-9 
Gaussian Network Model, 60-10 
Gaussian properties, 8-9 
Gauss-Jordan elimination, 1—7 to 1-9 
Gauss-Markov model 
least squares estimation, 53-11 
linear statistical models, 52-8 
Gauss-Markov theorem, 52-11 
Gauss multipliers, 38-7 
Gauss-Newton method, 51-8 
Gauss-Seidel algorithm, 41-14 
Gauss-Seidel methods, 41-3 to 41-4 
Gauss transformations 
Gauss elimination, 38—7 
sparse matrix factorizations, 40-4 
Gauss vectors, 38-7 
GCDD (greatest common divisor domain), 23-2 
GEBAL LAPACK subroutine, 43-3 
General Inverse Eigenvalue Problem (GIEP), 
34-8 
Generalized cycle, digraphs, 29-2 
Generalized cycle products, 29-5 to 29-6 
Generalized eigenvalue problem, 43-1 to 43-3 
Generalized eigenvalues and eigenvectors, 59-7 
Generalized Euclidean Parallel Postulate, 65-3 
Generalized inverse, 52—4 
Generalized Laplacian, 36-10 to 36-11 
Generalized least squares estimator, 52-8 
Generalized least squares problem, 39-1 
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Generalized line graphs, 28-4 
Generalized Minimal Residual (GMRES) 
convergence rates, 41—15 to 41-16 
Krylov space methods, 41-7, 41-9 to 41-11 
linear systems of equations, 49-13 
matrices function behavior, 16-11 
preconditioners, 41-12 
Generalized octonions, 69—4 
Generalized quarternions, 69-4 
Generalized Schur complement, 52-4 
Generalized sign pattern, 33-2 
Generalized Singleton bound, 61-12 
Generalized singular value decomposition (GSVD), 
15-12 
Generalized stars, 34-10 to 34-14 
Generalized variance, 52—3 
General linear group 
group representations, 68-1 
matrix group, 67-1 
General Schur Theorem, 43-2 
Generator matrix 
convolutional codes, 61-11 
linear block codes, 61-3 
Generators 
certain integral domains, 23-2 
linear independence and rank, 25-12 
polynomial, 61-7 
Geometry and geometric aspects 
affine spaces, 65-1 to 65-4 
eigenvalues and eigenvectors, 4-6 
Euclidean geometry, 66—1 to 66-15 
Euclidean spaces, 65—4 to 65-6 
fundamentals, 65-1 
least squares solutions, 39-4 to 39-5 
linear programming, 50-13 
matrix power asymptotics, 25-8 
max-plus eigenproblem, 25-6 
nonnegative and stochastic matrices, 9-2 
projective spaces, 65-6 to 65-9 
semidefinite programming, 51-5 
Geometry Rotations’ package, Mathematica software, 
73-4 
Gerhard, Jiirgen, 72-21 
Gersgorin discs, 14—5 to 14—7 
Gersgorin theorem 
generalized cycle products, 29-5 
pseudospectra, 16-3 
GGBAK LAPACK subroutine, 43—7 
GGBAL LAPACK subroutine, 43-7 
GGHRD LAPACK subroutine, 43-7 
GIEP (General Inverse Eigenvalue Problem), 34-8 
Gilbert Varshamov bound 
linear code classes, 61-10 
main linear coding problem, 61-6 
Givens algorithm, 46-5 to 46-6 
givens function, Matlab software, 42-9 
Givens QR factorization, 38-14 
Givens rotations 
orthogonalization, 38-13 to 38-14 
QR factorization, 39-9 
tridiagonalization, 42-5, 42-7 to 42-8 
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Givens transformation, 38-13 
Glide reflection, 65-5 
Global invariant manifolds, 56-19 
Glossary, G—1 to G40 
GMRES (Generalized Minimal Residual) 
convergence rates, 41-15 to 41-16 
Krylov space methods, 41-7, 41-9 to 41-11 
linear systems of equations, 49-13 
matrices function behavior, 16-11 
preconditioners, 41-12 
Golay codes, 61-8 
Goldman-Tucker theorem, 51-6 
Golub-Kahan singular value decomposition, 45-6 
Golub-Reinsch singular value decomposition, 45-6 
Gondran-Minoux properties, 25-12, 25-13 
Google (search engine) 
information retrieval, 63-10 to 63-14 
Markov chains, 54—4 to 54-5 
PageRank, 63-11 
Web search, 63-9 
Goppa code, algebraic geometric, 61-10 
grad, Mathematica software, 73-15, 73-16 
Graded algebras, 70-8 to 70-10 
Grading 
eigenvalue problems, 15-13 
Krylov subspaces, 49-6 
polar decomposition, 15-8 
singular value problems, 15-15 
tensor algebras, 13-20 to 13-22 
Gradual underflow, 37-11 to 37-12 
Gragg studies, 44-4 
Graham-Pollak theorem, 30-9 
Gramian, Euclidean simplexes, 66-8 
Gram matrices 
Euclidean geometry, 66-5 to 66-7 
Hermitian matrices, 8—1, 8—2 
Gram-Schmidt calculation, Maple software, 72-6 
Gram-Schmidt methods 
Arnoldi factorization, 44—3 to 44-4 
unitary similarity, 7-2, 7-4 
Gram-Schmidt orthogonalization, 5-8 to 5-10 
GramScmidt, Mathematica software, 73—4 
Graphical user interfaces (GUIs), 71-19 to 71-22 
Graphics, Matlab software, 71-14 to 71-17 
Graphics Arrow, Mathematica software, 73-5 
Graphs, see also Algebraic connectivity; Euclidean 
geometry; specific type of graph 
adjacency matrix, 28-5 to 28-7 
association schemes, 28-11 to 28-12 
doubly stochastic matrices, 27-10 
eigenvalues, 28-5 to 28-7 
fundamentals, 28—1 to 28-3 
matrix completion problems, 35-2 
matrix representations, 28-7 to 28-9 
modeling and analyzing fill, 40-11 
multiplicative D-stability, 19-6 
multiplicities and parter vertices, 34-2 
parameters, 28—9 to 28-11 
simplexes, 66-10 
special types, 28-3 to 28-5 
Grassmann, Taksar, Heyman (GTH) trick, 54-13 


Grassmann characteristics 
dynamical systems, 56-9 to 56-11 
Floquet theory, 56—13 
manifolds, dynamical systems, 56-7 
matrix pair, 15-12 
tensor algebras, 13-21 
tensors, 13-12 to 13-17 
Gray Code order, 31-12 
Greatest common divisor domain (GCDD), 23-2 
Greatest common divisors, 23—2 
Greatest integer function, P-3 
Greenbaum, Anne, 41-1 to 41-17 
Green’s functions, 59-10 to 59-11 
Griesmer bound, 61-5 
Grobman-Hartman theorem, 56—20 
Group, P-3 to P-4 
Group inverse, 9-2 
Group of invertible linear operators, 67-1 
Group representations 
characters, 68—5 to 68-6 
character table, 68-6 to 68-8 
fundamentals, 68—1 to 68-3 
induction of characters, 68-8 to 68-10 
matrix representations, 68-3 to 68-5 
orthogonality relations, 68-6 to 68-8 
restriction of characters, 68—8 to 68-10 
symmetric group representations, 68-10 to 68-11 
Group ring, 68-2 
Grover’s search algorithm, 62-15 to 62-17 
GSVD, see Generalized singular value decomposition 
(GSVD) 
GTH (Grassmann, Taksar, Heyman) trick, 54-13 
GUI, see Graphical user interfaces (GUIs) 
Gunaratne, Ajith, 60-13 
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Hacjan studies, 50-23 
Hadamard-Fischer inequality, 8-10 
Hadamard inequalities 
determinantal relations, 14—11 
inequalities, 17-11 
Hadamard matrix 
nonsquare case, 32-5 
permanents, 31-8 
square case, 32—2 
Hadamard product 
complex sign and ray patterns, 33-14 
positive definite matrices, 8-9 
rank and nullity, 14-14 
square matrices, 27-4 
totally positive and negative matrices, 21-10, 
21-11 
Hadamard’s determinantal inequality, 8-10, 8-11 
Haemers, Willem H., 28—1 to 28-12 
Half-line subset, 13-24 
Halfspaces, 25-11 
Hall, Frank J., 33-1 to 33-17 
Hall matrices, 27—3, 27-4 
Hamilton cycle, 28-1 
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Hamiltonian, minimally chordal symmetric, 
35-15 
Hamiltonian operator, 59-2 
Hamiltonian system, 56-13 
Hamilton-Jacobi partial differential equations, 
25-6 
Hamming association scheme, 28-12 
Hamming properties 
code, 61-6, 61-9 
distance, 61-2 
weight, 61-2 
Han, Lixing, 5-1 to 5-16 
Hankel matrices 
Maple software, 72-18 
structured matrices, 48-2, 48-3 
totally positive and negative matrices, 21-12 
HankelMatrix, Mathematica software, 73-6 
Hansen, Per Christian, 39-1 to 39-12 
Hard constraints, 51-1 
Hardware floats, Maple software, 72-14 
Hardy, Littlewood, Pélya theorem, 27-11 
Hardy space, 57-5 
Hare, Dave, 72-21 
Hartman-Grobman theorem, 56—20 
Hat matrix, 52-9 
Hautus-Popov test, 57-8 
Heat equation, 59-11 
Heat kernel, 59-11 
Height characteristic, 9-7, 26-8 
Hentzel, Irvin, 69-25 
Hereditary, 35-2 
Hermite normal form, 23—5 to 23-7, 23-6 
Hermitian characteristics 
angular momentum and representations, 59-9 
Arnoldi factorization, 44—3 
classical groups, 67-5 
differential-algebraic equations, 55-15 
differential equations, 55-11 
extensions, 16-13 
iterative solution methods, 41—4 to 41-7 
Jordan algebras, 69-13 
Kronecker products, 10-9 
Schrédinger’s equation, 59-7, 59-8 
Schur complements, 10-7 
splitting theorems and stability, 26-14 
Hermitian forms, 12—7 to 12-9 
Hermitian matrices 
adjoint, 1-4 
adjoint operators, 5-6 
Arnoldi process, 49-10 
bilinear forms, 12—7 to 12-9 
eigenvalues, 8-3 to 8-5, 15-5 to 15-6 
fundamentals, 1-4, 8-1 to 8-2 
inertia, 19-2 
multiplicities and Parter vertices, 34—2 
numerical range, 18-6 
order properties, eigenvalues, 8-3 to 8-5 
positive definite and semidefinite matrices, 
35-8 
positive definite characteristics, 5—2 
pseudo-inverse, 5-12 
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relative perturbation theory, 15-14 
singular value decomposition, 5-11 
singular values, 17-2, 17-13 to 17-14 
sparse matrices, 49-4 to 49-5 
spectral theory, 7-5, 7-8 
standard linear preserver problems, 22-5 
submatrices and block matrices, 10-2 
symmetric factorizations, 38-15 
Hermitian positive definite and semidefinite, ARPACK, 
76-7 to 76-8 
Hermitian preconditioning, 41-3 
Hermitian properties 
linear operators, 5-5 
pencils, 24—6 
property L, 24-6 
HermitianTranspose, Maple software, 72-3, 72-5 
Hershkowitz, Daniel, 19-1 to 19-10 
hes, Mathematica software, 73-16 
HessenbergDecomposition, Mathematica software, 
73-19, 73-27 
Hessenberg pattern, 33-3, 33-4 
Hessian properties 
Hermitian matrices, 8—2 
semidefinite programming, 51-10 
Hestenes studies, 46-2 
Hidden constraint, 51-10 
Higham, Nicholas J., 11-1 to 11-12 
Higham and Tisseur studies, 16-12 
Highest weight vector, 70-7 
High relative accuracy, eigenvalues and singular 
values 
accuracy, 46—2 to 46-5, 46-7 to 46-10 
fundamentals, 46-1 to 46-2 
one-sided Jacobi SVD algorithm, 46-2 to 46-5 
positive definite matrices, 46-10 to 46-14 
preconditioned Jacobi SVD algorithm, 46-5 to 
46-7 
rank revealing decomposition, 46-7 to 46-10 
structured matrices, 46-7 to 46-10 
symmetric indefinite matrices, 46-14 to 46-16 
High relative accuracy bidiagonal singular value 
decomposition, 45—7, 45-8 
hilb command, Matlab software, 71-18 
Hilbert matrix 
linear systems conditioning, 37-11 
preconditioned Jacobi SVD algorithm, 46-7 
rank revealing decomposition, 46-10 
HilbertMatrix, Mathematica software 
matrices, 73-6 
singular values, 73-18 
Hilbert-Schmidt inner product, 13-23, 13-24 
Hilbert spaces 
quantum computation, 62-2 
random signals, 64—4 
Schrédinger’s equation, 59-7 
Hill’s equation, 56-14 
Hirsch and Bendixson inequalities, 14-2 
HITS (Hypertext Induced Topic Search), 63-9 
HKM method, 51-8 
H-matrices, 19-9 
Hodge star operator, 13-24 to 13-26 
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Hoffman polynomial, 28-5, 28-6 
Hoffman-Wielandt inequality, 7-7 
Hoffman-Wielandt theorem, 15-2 
Hogben, Leslie, 6-1 to 6-14, 35-1 to 35-20 
Hélder inequality, 37-3 
Holder norm, 37-2 
Holmes, Randall R., 68—1 to 68-11 
Homogeneity 
cone programming, 51-2 
coordinates, 65-7 
differential equation, 2-3, 2-4 
Euclidean simplexes, 66-8 
function field, linear code classes, 61-10 
linear differential equations, 55-2 
line coordinates, 65-7 
Markov chains, 54—1 
partial inverse M-matrices, 35-14 
pencil strict equivalence, 23-9 
polynomials, 23-2, 23-9 
projective spaces, 65-7 
systems of linear equations, 1-9, 1-10 
tensor algebras, 13-21 
vector norms, 37—2 
vector seminorms, 37—3 
Homogeneous of degree, 13-21 
Homomorphism 
bimodules, 69-6 
modules, 70-7 
nonassociative algebra, 69-3 
Homotopy approach, 20-12 
Hopf algebra, 69-18 
Horn inequalities, 14-9 
Hotelling’s distribution, 53-9, 53-10 
Hotelling studies, 32-1 
Householder method 
algorithm efficiency, 37-17 
singular value decomposition, 45-5 
Householder properties 
algorithm, 46-5 to 46-6 
QR factorization, 38-13 
reduction, bidiagonal form, 45-5 
transformation, 38-13 
vectors, 38-13 
Householder reflections 
orthogonalization, 38-13, 38-15 
tridiagonalization, 42-6 
Howell, Kenneth, 58—1 to 58-20 
lhs, Mathematica software, 73-20 
HSEQR LAPACK subroutine, 43-6 
Hunacek, Mark, 65-1 to 65-9 
Hurwitz characteristics, 57—9 
Hurwitz command, Maple software, 72-20 
Hurwitz matrix, 57—7 
Hyperacute simplexes, 66-10 
Hyperbolic characteristics 
linear dynamical systems, 56-5 
linearization, 56-19 
symmetric indefinite matrices, 46-14 
Hyperbolic Jacobi, 46-15 
Hyperlink matrix, 63-10 
Hyperplanes, 66—2, 66-2 to 66-4 
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Hypotenuse, simplexes, 66-10 
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IAP (inertially arbitrary pattern), 33-11 
Ideal characteristics 

algebras, 69-18 

Lie algebras, 70-1 

nonassociative algebra, 69-3 
Idempotence 

general properties, 69-5 

invariant subspaces, 3-6 

nilpotence, 2-12 
Identity, linear transformations, 3-1 
Identity matrix 

fundamentals, 1-4, 1-6 

matrices, 1—4 

max-plus algebra, 25-1 

systems of linear equations, 1-13 
IdentityMatrix, Mathematica software 

decomposition, 73-19 

eigenvalues, 73-14 

matrices, 73-6, 73-8 

matrix algebra, 73-11 
Identity pattern, 33-2 
IEPs, see Inverse eigenvalue problems (IEPs) 
If, Mathematica software, 73-8 


IF-RCF (invariant factors rational canonical form) 


matrix, 6-12 
if statements, Matlab software, 71-11 
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IIR (infinite impulse response), 64-2, 64-3, 64-6 to 64-7 


Ill-conditioned properties 


conditioning and condition numbers, 37-7 


linear systems, 37-10 to 37-11 
symmetric indefinite matrices, 46-16 

Image, kernel and range, 3-5 

Immanant, 21—10, 31-13 

Implicitly restarted Arnoldi method (IRAM) 
Arnoldi factorization, 44—2 to 44-4 
convergence, 44—9 to 44-10 
fundamentals, 44—1 
generalized eigenproblem, 44-11 
implicit restarting, 44-6 to 44-8 
Krylov methods, 44-11 to 44-12 
Krylov subspace projection, 44—1 to 44-2 
polynomial restarting, 44-5 to 44-6 
restarting process, 44—4 to 44-5 
sparse matrices, 43-10 
spectral transformations, 44-11 to 44-12 
subspaces, 44—9 to 44-10 

Implicitly shifted QR method, 42-9 to 42-11 

Implicit QR iteration, 43-5 

Implicit restarting, 44-6 to 44-8 

Imprimitive digraphs, 29-9 

Imprimitive gate, 62-7 

Improper divisors, 23-2 

Improper hydroplane, 66-8 

Improper point, 66-8 

Impulse response, 64—2 
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imread command, Matlab software, 71-15 
Incidence matrix 
binary matrices, 31-5 
matrix representations, 28-7 
square case, 32-2 
Incomplete Cholesky decomposition, 41-11 
Incomplete LU decomposition, 41-11 
Inconsistency, systems of linear equations, 1-9 
Increment, 74—2 
Indecomposition, 70-4 
Indefinite matrices, 8-6 
Independence 
direct sum decompositions, 2-5 
graph parameters, 28-9 
Independent and identical distribution, 53-3 
Index 
Bezout domains, 23-9 
Collatz-Wielandt sets, 26—5 
doubly stochastic matrices, 27-10 
eigenvectors, 6-2 
linear differential-algebraic equations, 55-7 
polynomial interpolation, 11-2 
reducible matrices, 9-11 
Indexing module, 63-9 
Index of imprimitivity 
imprimitive matrices, 29-10 
irreducible matrices, 9-3 
reducible matrices, 9—7 
Index of primitivity, 29-9 
Index period, 58-3 
Indices, Web search, 63-9 
Indirect isometry, 65-5 
Individual condition number, 15-1, 15-10 
Induced bases, 2—5, 13-15 
Induced inner product, 13-23 
Induced subdigraphs, 29-2 
Induced subgraphs, 28-2 
Induction of characters, 68-8 to 68-10 
Inductive structure, 27—4 to 27-6 
Inequalities, see also Equalities and inequalities, 
matrices 
irreducible matrices, 9—5 
reducible matrices, 9-10 
singular values and singular value inequalities, 17—7 
to 17-12 
Inertia 
congruence, 8-5 
matrix stability and inertia, 19-2 to 19-3 
minimum rank, 33-11 to 33-12 
preservation, 19-5 
Inertia and stability 
additive D-stability, 19-7 to 19-8 
fundamentals, 19-1 to 19-2 
inertia, 19-2 to 19-3 
Lyapunov diagonal stability, 19-9 to 19-10 
multiplicative D-stability, 19-5 to 19-7 
stability, 19-3 to 19-5 
Inertially arbitrary pattern (IAP), 33-11 
Inertia set, 33-11 
Inexact preconditioners, 41-16 to 41-17 
Infeasibility, 50-1 
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Inference 
multivariate normal, 53—4 to 53-5 
statistical, 53-12 to 53-13 
Infinite dimensional, 2-3 
Infinite impulse response (IIR), 64-2, 64-3, 64-6 to 64-7 
Infinite Markov chain, 54-2 
Infinitesimal element, 21-11 
infinity, Maple software, 72-5 
Infinity, Mathematica software, 73-26 
Information retrieval 
fundamentals, 63-1 
Google’s page rank, 63-10 to 63-14 
latent semantic indexing, 63-3 to 63-5 
nonnegative matrix factorization, 63-5 to 63-8 
vector space method, 63-1 to 63-3 
Web search, 63-8 to 63-10 
Inheritance principle, 8-7 
Inhomogeneity 
Euclidean simplexes, 66-8 
linear differential equations, 55-2 
Initial minor data sets, 21-7 
Initial parameter settings, ARPACK, 76-6 
Initial state vector, 4-10 
Initial value problem, 55-2, 56-2 
Injective, kernel and range, 3-5 
inline command, Matlab software, 71-11 to 
71-12 
Inner distribution, 28-11 
Inner product and inner product spaces 
adjoints of linear operators, 5-5 to 5-6 
floating point numbers, 37-14 
fundamentals, 5-1 to 5-3 
multilinear algebra, 13-22 to 13-24 
orthogonality relations, 68—7 
Innovations process, 64—6 
Innovations representation, 64-6 
Input, algorithms and efficiency, 37-16 
Input space, 57-2 
Insert, Mathematica software, 73-13 
Instability, see also Stability 
error analysis, 37-18 to 37-21 
numerical methods, 37—18 to 37-21 
Integers, Mathematica software, 73-24 
Integral domains, matrices over 
Bezout domains, 23-8 to 23-9 
certain integral domains, 23-1 to 23-4 
fundamentals, 23-1 
linear equations, 23-8 to 23-9 
matrix equivalence, 23—4 to 23-8 
strict equivalence, pencils, 23-9 to 23-10 
Integral kernels, 59-10 
Integrate, Mathematica software, 73-1 
Interactive software 
freeware, 77-1 to 77-3 
Maple, 72-1 to 72-21 
Mathematica, 73—1 to 73-27 
Matlab, 71-1 to 71-22 
Interchange 
classes of (0,1)-matrices, 27—7 
tournament matrices, 27-9 
interface, Maple software, 72-7 
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Interior point methods, 50-23 to 50-24 
Interior solution, 50-23 
Interlaces, P—4 
Interlacing inequalities, 34—2 
Interlacing inequalities, 8—3 to 8-4 
Internal direct sum, 69-3 
Internal stability, 57-15, see also Stability 
International Standard Book Number (ISBN) code, 
61-4 
Intersection, graphs, 28-2 
Intersection number, 28-11 
Intraclass correlation, 52-9 
Invariant factors 
canonical forms, 6-12 to 6-14 
inverse eigenvalue problems, 20-3 
matrix equivalence, 23-5 
Invariant factors rational canonical form (IF-RCF) 
matrix, 6-12 
Invariant manifold theorem, 56—20, 56-21 
Invariant polynomials 
inverse eigenvalue problems, 20-3 
matrix equivalence, 23-5 
Invariants 
face, 26-2 
group representations, 68-1 
probability vectors, 54-2 
random linear dynamical systems, 56-14 
simultaneous similarity, 24-8 
subspaces, projections, 3-6 to 3-7 
vectors, 54—2 
inv command, Matlab software, 71-17 
Inverse, Maple software, 72-13 
Inverse, Mathematica software 
eigenvalues, 73-16 
linear systems, 73-20, 73-22 
matrices, 73-6, 73-8 
matrix algebra, 73-10, 73-11 
Inverse, random vectors, 52—4 
Inverse discrete Fourier transform (inverse DFT), 
58-9 
Inverse eigenvalue problems (IEPs) 
affine parameterized IEPs, 20-10 to 20-12 
fundamentals, 20-1 
nonnegative matrices, 9-22, 20-5 to 20-10 
nonzero spectra, nonnegative matrices, 20-7 to 
20-8 
numerical methods, 20-11 to 20-12 
prescribed entries, 20-1 to 20-3 
spectra, nonnegative matrices, 20-6 to 20-10 
trees, 34—8 
2x2 block type, 20-3 to 20-5 
Inverse Fourier transform 
Fourier analysis, 58-2 
Karle-Hauptman matrix, 60-8 
Inverse iteration, 42—2, 42-14 to 42-15 
Inverse Lanczos methods, 16-11 
Inverse matrices, 1-12 
InverseMatrixNorm, Mathematica software, 73-10 
Inverse M-matrices 
completion problems, 35-14 to 35-15 
fundamentals, 9-17 
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Inverse nonnegatives 
inverse patterns, 33-12 
M-matrices, 9-17 
splitting theorems and stability, 26-13 
Inverse patterns, 33-12 to 33-14 
Inverse-positivity matrices, 9-17 
Inverse sign-pattern matrices, 33-12 to 33-14 
Inverse tridiagonal matrices, 21-3 
Inversion 
fast matrix multiplication, 47-9, 47-9 to 47-10 
isomorphism, 3—7 
matrix equalities and inequalities, 14-15 to 14-17 
Inverted file, Web search, 63-9 
Invertibility 
fundamentals 
isomorphism, 3—7 
matrix equivalence, 23-5 
Invertible linear operators, 67-1 
Invertible matrices 
decomposable tensors, 13-7 
fundamentals, 1-12 
LU factorizations, 1-15 
Invertible Matrix Theorem, 1—12 
Invert spectral transformation mode, ARPACK, 76-7 
inv function, Matlab software, 71-3 
Involution, 69-2 


IRAM, see Implicitly restarted Arnoldi method (IRAM) 


Irreducibility 

bimodules, 69-6 

certain integral domains, 23-2 

characterizing, 9-3 

characters, 68—5, 68-6 

group representations, 68-1 

imprimitive matrices, 29-9 to 29-11 

matrix group, 67-1 

matrix representations, 68-3 

max-plus algebra, 25-2 

modules, 70—7 

Perron-Frobenius theorem, 26—2 

simultaneous similarity, 24-8 
Irreducible classes, 54—5 to 54—7 
Irreducible components, 27-5 to 27-6 
Irreducible matrices 

balancing, 43-4 

digraphs, 29-6 to 29-8 

fundamentals, 9-2 to 9-7 

nonnegative matrices, 9-2 to 9-7 


ISBN (International Standard Book Number) code, 61—4 


Isolated invariant, 56-7 
Isolated vertex, 28-2 
Isometry 
classical groups, 67-5 
Euclidean spaces, 65—4 
Isomorphism 
graphs, 28-1 
Grassmannian manifolds, 56-10 
group representations, 68-1 
matrix representations, 68-3 
nonassociative algebra, 69-3 
nonsingularity characterization, 3—7 to 3-8 
Isoperimetric number, 28-9 


1-24 


Isotalo, Jarkko, 53-14 

Iterative methods 
function computation methods, 11-10 
Krylov subspaces and preconditioners, 41-2 
Toeplitz solvers, 48-5 

Iterative solution methods, linear solutions 
CG convergence rates, 41-14 to 41-15 
convergence rates, 41-14 to 41-16 
error estimation, 41-16 to 41-17 
finite precision arithmetic, 41-16 to 41-17 
fundamentals, 41-1 to 41-2 
GMRES convergence rates, 41-15 to 41-16 
Hermitian problems, 41-4 to 41-7 
inexact preconditioners, 41-16 to 41-17 
Krylov subspaces, 41-2 to 41-4 
MINRES convergence rates, 41-14 to 41-15 
non-Hermitian problems, 41-7 to 41-11 
non-optimal Krylov space methods, 41-7 to 

41-11 

optimal Krylov space methods, 41—4 to 41-11 
preconditioned algorithms, 41-12 to 41-14 


preconditioners, 41-2 to 41-4, 41-11 to 41-12 


stopping criteria, 41-16 to 41-17 


J 


Jacobian characteristics 

linearization, 56-20 

nonassociative algebra, 69-2 
Jacobi-Davidson methods, 43-10 to 43-11 
Jacobi identity 

Akivis identity, 69-19 

Lie algebras, 70-2 

nonassociative algebra, 69-2 
Jacobi matrices, 21-4 
Jacobi methods 


Krylov subspaces and preconditioners, 41-3 to 


41-4 


symmetric matrix eigenvalue techniques, 42-17 to 


42-19 
Jacobi rotation 
Jacobi method, 42-17 


singular value decomposition, 45-11 to 45-12 


Jacobi’s Theorem, 4—5 
Jeffrey, David J., 72-1 to 72-21 
Johnson, Charles R., 34-1 to 34-15 
Johnson association scheme, 28-12 
Join, graphs, 28-2 
Join, Mathematica software 
matrices manipulation, 73-13 
vectors, 73-3 
Jordan algebras 
computational methods, 69-21 
nonassociative algebra, 69-3, 69-12 to 69-14 
power associative algebras, 69-15 
Jordan basis, 6—3, 6—4, 6-5 
Jordan blocks 
Jordan canonical form, 6-3 
K-reducible matrices, 26-9 
linear differential equations, 56-4 
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matrix function, 11-4 

matrix product, 18-9 
Perron-Schaefer condition, 26-7 
simultaneous similarity, 24-10 


Jordan canonical form, see also real-Jordan canonical 


form 
canonical forms, 6—3 to 6-6 
differential equations, 55-6, 55-8 
matrix function, 11-1 to 11-2, 11-4 
reducible matrices, 9-8 
unitary similarity, 7-3 
Jordan characteristics 
chain, 6—4 
form matrices, 56-3 
identity, nonassociative algebra, 69-3 
invariants, 6-3, 6-4 
product, nonassociative algebra, 69-3 
structures, simultaneous similarity, 24-9 
jordan command, Matlab software, 71-9 
JordanDecomposition, Mathematica software, 
73-18 
JordanForm, Maple software, 72-11, 72-12 
Jordan-Wielandt matrix, 17-2 
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Kahan-Cao-Xie-Li theorem, 15-3 
Kahan matrix, 17-4 
Kalman-Bucy filter 

LTI systems, 57-17 

state estimation, 57—11, 57-12, 57-13 
Kalman decomposition, 57—7 to 57-8 
Kalman filter, 57-13 
Kamm and Nagy studies, 48-9 
Kantorovich Inequality, 52-10 
Kapranov rank, 25-13 
Karle-Hauptman matrix, 60-7 to 60-9 
Karp’s formula 

maximal cycle mean, 25-5 

max-plus eigenproblem, 25-7 
Kaufman studies, 44—4 
Kernel 

Bezout domains, 23-8 

characters, 68-5 

group representations, 68-2 

least squares solution, 39-4 

linear independence, span, and bases, 2-6 

linear inequalities and projections, 25-10 

matrix representations, 68-3 

range, 3—5 to 3-6 
kernel, Mathematica software, 73-1 
Killing form 

Malcev algebras, 69-16, 69-17 

semisimple and simple algebras, 70-3 
Kinetic energy 

Lagrangian mechanics, 59-5 

oscillation modes, 59-2 
Kingman’s inequality, 25-5 
Kleene star, 25-2, 25-3 
Kliemann, Wolfgang, 6-14, 56-1 to 56-21 
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Klienberg, Jon, 63-9 
Klyachko studies 
eigenvalues, 17-13 
Hermitian matrices, 8—4 
Knott Inequality, 52-10 
Knutson and Tao studies 
eigenvalues, 17-13 
Hermitian matrices, 8—4 
Koteljanskii’s Determinantal Inequality, 8-10 
Kravcuk polynomials, 28-12 
K-reducible matrices, 26-8 to 26-13 
Krein condition, 28-11 
Kreiss Matrix theorem, 16-10 
Kronecker canonical form, 55—7 to 55-10 
Kronecker product 
linear maps, 13-9 
Matlab software, 71-4 
matrix similarities, 24-1 
partitioned matrices, 10-8 to 10-9 
positive definite matrices, 8-10 
rank and nullity, 14-13 
semidefinite programming, 51-3, 51-4 
structured matrices, 48—4 
tensor products, 13-8 
Kronecker symbol, 28-11 
kron function, Matlab software, 71—4 
Krylov characteristics 
implicitly restarted Arnoldi method, 44-11 to 44-12 
matrix, Lanczos method, 42-19 
sequence, 49-5 
spaces, 41—2 
vectors, 42—20 
Krylov-Schur algorithm, 43-9 
Krylov subspaces 
eigenvalue computations, 49-12 
iterative solution methods, 41-2 to 41-4 
large-scale matrix computations, 49-5 to 49-6 
projection, 44-1 to 44-2 
k-step, 44-3 
Ky-Fan norms 
polar decomposition, 15-8 
unitarily invariant norms, 17-5, 17-6 
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Lagrangian function, 51-5 

Lagrangian mechanics, 59-5 to 59-6 

Lagrangian properties 
multiplier, differential equations, 55-3 
primal-dual interior point algorithm, 51-8 
relaxation, 51-10 

Laguerre polynomial, 31-10 

Lanczos algorithm 
Krylov space methods, 41-4 to 41-5, 41-10 
symmetric matrix eigenvalue techniques, 42-19 to 

42-21 

Lanczos matrices 
eigenvalue computations, 49-12 
linear systems of equations, 49-13 
symmetric Lanczos process, 49-7 
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Lanczos methods 
implicit restarting, 44-6 
pseudospectra computation, 16-11 
total least squares problem, 48-9 
Lanczos process, 49-10 
Lanczos vectors 
Arnoldi factorization, 44-3 
nonsymmetric Lanczos process, 49-8 
symmetric Lanczos process, 49-7 
Landaw’s theorem, 27-9 
Langou, Julien, 74-1 to 74-7, 75-1 to 75-23, 77-1 to 
77-3 
Langville, Amy N., 63-1 to 63-14 
LAPACK subroutine package 
dense matrices, 43-3 
DLARGV subroutine, 42-9 
DLARTV subroutine, 42-9 
DLAR2V subroutine, 42-9 
DORGTR, 42-8 
DORGTR subroutine, 42-8 
DSBTRD subroutine, 42-9 
DSTEBZ subroutine, 42-15 
DSTEDC subroutine, 42-14 
DSTEGR subroutine, 42-17 
DSTEIN subroutine, 42-15 
DSYEV subroutine, 42-11 
DSYTRD, 42-8 
DSYTRD subroutine, 42-8 
equality-constrained least squares problems, 75-6 to 
75-7 
execution times, 42-22 
fundamentals, 75-1 to 75-2 
GEBAL subroutine, 43-3 
GGBAK subroutine, 43—7 
GGBAL subroutine, 43—7 
GGHRD subroutine, 43-7 
HSEQR subroutine, 43-6 
least squares problems, 75-4 to 75-7 
linear equality-constrained least squares problems, 
75-6 to 75-7 
linear least squares problems, 75-4 to 75-6 
linear model problem, 75-8 to 75-9 
linear system of equations, 75—2 to 75-4 
models, 75-8 to 75-9 
nonsymmetric eigenproblems, 75-17 to 75-20 
nonsymmetric eigenvalue problem, 75-11 to 75-13 
singular value decomposition, 75-13 to 75-15, 75-20 
to 75-23 
symmetric definite eigenproblems, 75-15 to 75-17 
symmetric eigenvalue problem, 75-9 to 75-11 
TGEVC subroutine, 43—7 
TGSEN subroutine, 43-7 
TGSNA subroutine, 43—7 
TREVC subroutine, 43-6 
TREXC subroutine, 43—7 
tridiagonalization, 42-8 
TRSEN subroutine, 43—7 
TRSNA subroutine, 43-7 
Laplace expansion 
determinantal relations, 14-10 
determinants, 4—1, 4-5 
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multiplication, 13-18 to 13-19 
permanents, 31-2 
permanents evaluation, 31-12 
Laplace transforms 
frequency-domain analysis, 57—6 
LTI systems, 57-14 
Laplacian matrices 
algebraic connectivity, 36-1, 36-2 
Fiedler vectors, 36—7, 36-8 to 36-9 
Hermitian matrices, 8—5 
Laplacian properties 
algebraic connectivity, 36-10 to 36-11 
eigenvalues, 28-7 
graph parameters, 28-9, 28-10 
graphs, 28-8 to 28-9 
matrix representations, 28-7 
Large-scale matrix computations 
Arnoldi process, 49-10 to 49-11 
dimension reduction, 49-14 to 49-15 
eigenvalue computations, 49-12 
fundamentals, 49-1 to 49-2 
Krylov subspaces, 49-5 to 49-6 
linear dynamical systems, 49-14 to 49-15 
linear systems, equations, 49-12 to 49-14 
nonsymmetric Lanczos process, 49-8 to 49-10 
sparse matrix factorizations, 49-2 to 49-5 
symmetric Lanczos process, 49-6 to 49-7 
Laser method, 47-8, 47-9 
Last, Mathematica software 
matrices manipulation, 73-13 
singular values, 73-18 
vectors, 73-3 
Latent semantic indexing (LSI), 63-3 to 63-5 
Latin rectangle, 31-6 
Lawley-Hotelling trace statistic, 53-13 
LDPC (low density parity check) codes, 61-11 
LDU factorization, 1-14 to 1-15 
Leading diagonal, 15-12 
Leading dimension, 74-2 
Leading entry, 1-7 
Leading principle matrices, 1-6, 1-15 
Leading principle minors 
determinants, 4—3 
recognition and testing, 21-7 
stability, 19-3 
Leading principle submatrices, 1-4 
Leal Duarte, Anténio, 34—1 to 34-15 
Least squares algorithms, 39-6 to 39-7 
Least squares estimation 
linear statistical models, 52-8 
multivariate statistical analysis, 53-11 to 53-12 
Least squares problems 
fundamentals, 5-14 to 5-16 
LAPACK subroutine package, 75—4 to 75-7 
Matlab software, 71—7 to 71-9 
numerical stability and instability, 37-20 
Least squares solutions, linear systems 
algebraic aspects, 39-4 to 39-5 
algorithms, 39-6 to 39-7 
damped least squares, 39-9 to 39-10 
data fitting, 39-3 to 39-4 


Handbook of Linear Algebra 


downdating, 39-8 to 39-9 
fundamentals, 39-1 to 39-3 
geometric aspects, 39-4 to 39-5 
orthogonal factorizations, 39-5 to 39-6 
QR factorization, 39-8 to 39-9 
rank revealing decompositions, 39-11 to 39-12 
sensitivity, 39-7 to 39-8 
updating, 39-8 to 39-9 
Lebesgue spaces, 57-5 
Lee and Seung algorithm, 63-6, 63-7 
Left alternative algebra, 69-10 
Left alternative identities, 69-2 
Left deflating subspaces, 55-7 
left divide operator, Matlab software, 71-7 
Left eigenvector, 4-6 
Left Kronecker indices, 55—7 
Left Krylov subspace, 49-8 
Left Lanczos vectors, 49-8 
Left-looking methods, 40-10 
Left Moufang identity, 69-10 
Left multiplication operators, 69-5 
Left-normalized product, 69-16 
Left preconditioning 
BiCGSTAB algorithm, 41-12 
Krylov subspaces and preconditioners, 41-3 
Left reducing subspaces, 55-7 
Left regular representation, 68-2 
Left singular space, 45-1 
Left singular vectors, 5-10, 45-1 
Legs, simplexes, 66-10 
Length, Mathematica software 
fundamentals, 73-27 
matrices, 73-6, 73-7 
matrix algebra, 73-12 
vectors, 73-3, 73-4 
Length characteristics 
Euclidean point space, 66-2 
graphs, 28-1 
max-plus algebra, 25-2 
sign-pattern matrices, 33-2 
stars, 34-10 
length command, Matlab software, 71-2, 71-9 
Length of a walk 
digraphs, 29-2 
graphs, 28-1 
Leon, Steven J., 71-1 to 71-22 
Level characteristic, 26-8 
Levinson-Durbin algorithm, 64-8, 64-9 
Lévy-Desplanques theorem, 14-6 
Li, Chi-Kwong, 18-1 to 18-11 
Li, Ren-Cang, 15-1 to 15-16 
Lie algebras 
angular momentum and representations, 59-10 
fundamentals, 70-1 to 70-3 
graded algebras, 70-8 to 70-10 
modules, 70—7 to 70-10 
nonassociative algebra, 69-2, 69-3 
semisimple algebras, 70-3 to 70-7 
simple algebras, 70-3 to 70-7 
Liénard-Chipart Stability Criterion, 19-4 
Likelihood function, 53-4 
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Likelihood ratio statistic, 53-13 LinearAlgebra package, Maple, 72-1, 72-2 
Limits, nonnegative and stochastic matrices, 9-2 LinearAlgebra’Tridiagonal’, Mathematica software, 
Limit set, chain recurrence, 56—7 73-20 
linalg package, Maple, 72-1 Linear block codes, 61-3 to 61-4 
Linder, David, 72-21 Linear characters, 68—5 
Line, Mathematica software, 73-5 Linear code classes, 61-6 to 61-11 
Linear algebra Linear combination 
adjoints of linear operators, 5-5 to 5-6 Euclidean point space, 66-2 
annihilator, 3-8 to 3-9 span and linear independence, 2-1 
bases, 2-10 to 2-12, 3-4 Linear constraints, 50-1 
change of basis, 2-10 to 2-12, 3-4 Linear dependence, 2-1, 66-2 
coordinates, 2-10 to 2-12 Linear differential-algebraic equations, 55-7 to 55-10, 
determinants, 4—1 to 4-6 55-14 to 55-16 
dimension theorem, 2-6 to 2-9 Linear differential equation of order, 55-1 
direct sum decompositions, 2-4 to 2-6 Linear differential equations 
eigenvalues and eigenvectors, 4—6 to 4-11 differential equations, 55-1 to 55-5 
Gaussian elimination, 1—7 to 1-9 dynamical systems, 56—2 to 56—4 
Gauss-Jordan elimination, 1—7 to 1-9 numerical methods, 54—12 
Gram-Schmidt orthogonalization, 5-8 to 5-10 Linear dynamical systems, 49-14 to 49-15, 56-5 to 56-7, 
idempotence, 2-12 see also Dynamical systems; Linear systems 
inner product spaces, 5—1 to 5-3, 5-5 to 5-6 Linear equality-constrained least squares problems, 
invariant subspaces, 3-6 to 3-7 75-6 to 75-7 
isomorphism, 3-7 to 3-8 Linear equations 
kernel, 3—5 to 3-6 cone invariant departure, matrices, 26-11 to 26-12 
least-squares problems, 5-14 to 5-16 matrices over integral domains, 23-8 to 23-9 
linear functionals, 3-8 to 3-9 systems of linear equations, 1-9 
linear independence, 2—1 to 2-3 LinearEquationsToMatrices, Mathematica, 73-20, 
linear transformations, 3—1 to 3-9 73-22 
LU factorization, 1-13 to 1-15 Linear form 
Maple software, 72-12 to 72-13 linear functionals and annihilator, 3-8 
matrices, 1-3 to 1-6, 1-11 to 1-13, 2-6 to 2-9, 3-3 to linear programming, 50-1 
3-4 Linear functionals, annihilator, 3-8 to 3-9 
nilpotence, 2-12 Linear hull, 66-2 
nonsingularity characterizations, 2-9 to 2-10, 3-7 to Linear independence and rank 
3-8 Euclidean point space, 66-2 
null space, 2-6 to 2-9 max-plus algebra, 25-12 to 25-14 
orthogonality, 5-3 to 5-10 span, 2-1 to 2-3 
orthogonal projection, 5-6 to 5-8 Linear inequalities and projections, 25-10 to 25-12 
projections, 3-6 to 3-7 Linearization 
pseudo-inverse, 5-12 to 5-14 alternative algebras, 69-11 
QR factorization, 5-8 to 5-10 dynamical systems, 56-19 to 56-21 
range, 3-5 to 3-6 Jordan algebras, 69-12 
rank, 2-6 to 2-9 Linear least squares problems, 75—4 to 75-6 
similarity, 3-4 Linear maps, 3-1, 13-8 to 13-10, see also Linear 
singular value decomposition, 5-10 to 5-12 transformations 
span, 2-1 to 2-3 Linear matrix groups, 67-3 to 67-5 
systems of linear equations, 1-9 to 1-11 Linear matrix similarities, 24—1 
transformations, linear, 3—1 to 3-9 Linear model problem, 75-8 to 75-9 
vectors, 1-1 to 1-3, 2—3 to 2—4, 3-2 to 3-3 Linear operators 
LinearAlgebra’Cholesky’, Mathematica software, 73-27 linear transformations, 3—1 
LinearAlgebra MatrixManipulation, Mathematica nonassociative algebra, 69-3 
software Linear ordinary differential equations 
decomposition, 73-19 fundamentals, 55-5 to 55-6 
fundamentals, 73-26 stability, 55-10 to 55-14 
linear systems, 73-22 Linear output feedback, 57-7 
LinearAlgebra’MatrixManipulation’ package, Linear prediction, 64—7 to 64-9 
Mathematica software, 73-2 Linear preserver problems 
LinearAlgebra[Modular], Maple software, 72-12 to additive preservers, 22-7 to 22-8 
72-13 fundamentals, 22-1 to 22-2 
LinearAlgebra Orthogonalization’ package, multiplicative preservers, 22-7 to 22-8 
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problems, 22—4 to 22-7 fundamentals, 41-1 to 41-2 
standard forms, 22-2 to 22-4 GMRES convergence rates, 41-15 to 41-16 
Linear programming Hermitian problems, 41-4 to 41-7 
canonical forms, 50—7 to 50-8 inexact preconditioners, 41-16 to 41-17 
duality, 50-13 to 50-17 Krylov subspaces, 41-2 to 41-4 
formulation, 50-3 to 50-7 MINRES convergence rates, 41-14 to 41-15 
fundamentals, 50-1 to 50-2 non-Hermitian problems, 41-7 to 41-11 
geometric interpretation, phase 2, 50-13 non-optimal Krylov space methods, 41-7 to 41-11 
interior point methods, 50-23 to 50-24 optimal Krylov space methods, 41-4 to 41-11 
linear approximation, 50-20 to 50-23 preconditioned algorithms, 41-12 to 41-14 
Mathematica software, 73-23 to 73-24 preconditioners, 41-2 to 41-4, 41-11 to 41-12 
matrix games, 50-18 to 50-20 stopping criteria, 41-16 to 41-17 
parametric programming, 50-17 to 50-18 Linear systems, least squares solutions 
phase 2 geometric interpretation, 50-13 algebraic aspects, 39-4 to 39-5 
pivoting, 50-10 to 50-11 algorithms, 39-6 to 39-7 
sensitivity analysis, 50-17 to 50-18 damped least squares, 39-9 to 39-10 
simplex method, 50-11 to 50-13 data fitting, 39-3 to 39-4 
standard forms, 50-7 to 50-8 downdating, 39-8 to 39-9 
standard row tableaux, 50-8 to 50-10 fundamentals, 39-1 to 39-3 
LinearProgramming, Mathematica software, 73-24 geometric aspects, 39-4 to 39-5 
Linear programs, 50-2 orthogonal factorizations, 39-5 to 39-6 
Linear-quadratic Gaussian (LQG) problem, 57-14 QR factorization, 39-8 to 39-9 
Linear-quadratic regulator (LQR) problem, 57-14 rank revealing decompositions, 39-11 to 39-12 
Linear representation, 68-1 sensitivity, 39-7 to 39-8 
Linear semidefinite programming, 51-3 updating, 39-8 to 39-9 
Linear skew product flows, 56-11 to 56-12 Linear systems of equations 
LinearSolve, Maple software, 72-9 LAPACK subroutine package, 75-2 to 75-4 
LinearSolve, Mathematica software large-scale matrix computations, 49-12 to 49-15 
decomposition, 73-19 Linear time-invariant (LTI) systems 
linear systems, 73-20, 73-21, 73-22 analysis, 57—7 to 57-10 
Linear state feedback, 57—7 control theory, 57-2 
Linear statistical models signal processing, 64—1 
fundamentals, 52-8 to 52-15 Linear transformations 
random vectors, 52—1 to 52-8 Euclidean spaces, 65—4 
Linear subspaces, 66-2 linear algebra, 3-1 to 3-9 
Linear system perturbations, 38-2 to 38-5 Linear unbiased estimator, 52-9 
Linear systems Linear zero function, 52-9 
conditioning, error analysis, 37-9 to 37-11 Line graphs, 28-4 
dynamical, 49-14 to 49-15 Lines 
equations, 49-12 to 49-15 affine spaces, 65-2 
fundamentals, 1-9 Euclidean point space, 66-2 
large-scale matrix computations, 49-12 to matrices, 27-2 
49-15 projective spaces, 65-6 
Mathematica software, 73-20 to 73-23 segments, 50-13, 65-2 
Matlab software, 71-7 to 71-9 Linklesly embedded graphs, 28—4 
structured matrices computations, 48-5 to ListDensityPlot, Mathematica software, 73-27 
48-8 L-matrices 
Linear systems, direct solution sign-pattern matrices, 33-5 to 33-7 
fundamentals, 38—1 sign solvability, 33-5 
Gauss elimination, 38—7 to 38-12 L-module, 70-7 
LU decomposition, 38-7 to 38-12 Local invariant polynomials, 23-9 
orthogonalization, 38-13 to 38-15 Local Perron-Schaefer conditions, 26—6 
perturbations, 38-2 to 38-5 Local similarity, 22-7, 24-1 
QR decomposition, 38-13 to 38-15 Local spectral radius, 26-2 
symmetric factorizations, 38-15 to 38-17 Local stable/unstable manifold, 56-19 
triangular linear systems, 38-5 to 38-7 Location, numerical range, 18—4 to 18-6 
Linear systems, iterative solution methods Loewner ordering, 8-10 
CG convergence rates, 41—14 to 41-15 Loewner partial ordering, 52-4 
convergence rates, 41-14 to 41-16 Loewy, Raphael, 12-1 to 12-9 
error estimation, 41-16 to 41-17 Log, Mathematica software, 73-26 


finite precision arithmetic, 41-16 to 41-17 Logarithms, 11-6 
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Log-barrier problem, 51-8 
Logical operators, Matlab software, 71-12 
Look-ahead strategies, 40-17, 41-10 
Loop digraphs, 29-2 
Loop graphs, 28-1 
Lorentz cone, 51—2 
Lovasz parameter, 28-10 
Low density parity check (LDPC) codes, 61-11 
Lower bounds, 47-8 
Lower Collatz-Wielandt numbers, 26—4 
LowerDiagonalMatrix, Mathematica software, 
73-6 
Lower Hessenberg matrices, 10-4 
Lowering operator, 59-8, 59-9 
Lower shift matrices, 48—1 
Lower triangular linear matrix, 38-5 
Lower triangular matrices, 1-4, 10-4 
Léwner partial ordering, 52-4 
LQG (linear-quadratic Gaussian) problem, 57-14 
LQR (linear-quadratic regulator) problem, 57-14 
LSI, see Latent semantic indexing (LSI) 
LTI (linear time-invariant) systems 
analysis, 57—7 to 57-10 
control theory, 57-2, 57—7 to 57-10 
signal processing, 64—1 
LUBackSubstitute, Mathematica software, 73-19 
LUBackSubstitution, Mathematica software, 73-20, 
73-22 
LU decomposition 
direct solution of linear systems, 38-7 to 38-12 
incomplete LU decomposition, 41-11 
LUDecomposition, Maple software, 72-9, 72-10 
LUDecomposition, Mathematica software 
decomposition, 73-18, 73-19 
linear systems, 73-20, 73-22 
Luenberger observer, 57-11, 57-12, 57-13 
LU factorizations 
Gaussian Eliminations, 1-13 to 1-15 
sparse matrix factorizations, 49-3 
LUMatrices, Mathematica software, 73-19 
Lyapunov equation 
differential-algebraic equations, 55-15 to 55-16 
differential equations, 55-11 
matrix equations, 57-10 
semidefinite programming, 51-9 
Lyapunov exponents 
Floquet theory, 56-13 
linear differential equations, 56—2 
linearization, 56-21 
linear skew product flows, 56-11 
random linear dynamical systems, 56-15, 56-16 
Lyapunov properties 
diagonal stability, 19-9 to 19-10 
scaling factor, 19-9 
spectrum, 56-16, 56-17 
stability, 55-10, 55-14 
Lyapunov spaces 
Grassmannian and flag manifolds, 56-10, 56-11 
linear differential equations, 56—2 
random linear dynamical systems, 56—15 
Lyapunov Stability Criterion, 19-4 
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Machado, Armando, 13-1 to 13-26 
Machine epsilon, 37-12 
Mackey’s Subgroup theorem, 68-9 
Main angles, adjacency matrix, 28-5 
Main Coding Theory Problem, 61-5 
Main diagonal, matrices, 1-4 
Main Linear Coding Problem, 61—5 to 61-6, 61-12 
Majorization, P-4 
Malcev algebras, 69-2, 69-3, 69-16 to 69-17 
Malcev module, 69-16 
Malcev theorem, 67-2 
M- and Mp- matrices, 35-12 to 35-13 
Manipulation of matrices, 73-13 to 73-14 
Mantissa, 37-11 
Map, Mathematica software 
decomposition, 73-19 
eigenvalues, 73-16 
fundamentals, 73-26 
matrices, 73-7 
matrix algebra, 73-10 
singular values, 73-17 
Maple software 
arrays, 72-8 to 72-9 
canonical forms, 72-15 to 72-16 
eigenvalues and eigenvectors, 72-11 to 72-12 
equation solving, 72-9 to 72-11 
functions of matrices, 72-19 to 72-20 
fundamentals, 72—1 to 72-2 
linear algebra, 72-12 to 72-13 
matrices, 72—4 to 72-7 
matrix factoring, 72-9 to 72-11 
modular arithmetic, 72-12 to 72-13 
numerical linear algebra, 72-13 to 72-15 
stability of matrices, 72-20 to 72-21 
structured matrices, 72-16 to 72-18 
vectors, 72—2 to 72—4 
Mapping theorems, 16-2 
MapThread, Mathematica software, 73-13 
Markov chains 
censoring, 54-11 
doubly stochastic matrices, 27-10 
eigenvalues and eigenvectors, 4-10 
finite, 54—9 to 54-11 
fundamentals, 54—1 to 54-5 
irreducible classes, 54—5 to 54—7 
numerical methods, 54—12 to 54-14 
Perron-Schaefer condition, 26—7 
states classification, 54—7 to 54-9 
Markov decision process, deterministic, 25-3 
Markowitz cost, 40-17 
Markowitz scheme, 40-18 
Maschke’s theorem 
group representations, 68-2 
matrix groups, 67-2 
matrix representations, 68-4 
Matching, bipartite graphs, 30-2 
Mathematical physics 
angular momentum, 59-9 to 59-10 
fundamentals, 59-1 to 59-2 
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Green’s functions, 59-10 to 59-11 
Lagrangian mechanics, 59-5 to 59-6 
oscillation modes, 59-2 to 59-5 
rotation group, representations, 59-9 to 
59-10 
Schrédinger’s equation, 59-6 to 59-9 
Mathematical programming, 50-1 
Mathematica software 
appendix, 73-25 to 73-27 
decompositions, 73-18 to 73-19 
eigenvalues, 73-14 to 73-16 
fundamentals, 73-1 to 73-2, 73-25 to 73-27 
linear programming, 73-23 to 73-24 
linear systems, 73-20 to 73-23 
manipulation of matrices, 73-13 to 73-14 
matrices, 73-5 to 73-9 
matrix algebra, 73-9 to 73-12 
singular values, 73-16 to 73-18 
vectors, 73-3 to 73-5 
Mathias, Roy, 17-1 to 17-15 
Mathieu equation, 56-14 
Matlab software 
bucky command, Matlab, 71-11 
built-in functions, 71-4 to 71-5 


built-in functions, Matlab software, 71—4 to 71-5 


cat command, Matlab, 71-2 

circul function, Matlab, 71-6 
colormap command, Matlab, 71-15 
colspace command, Matlab, 71-17 
contour command, Matlab, 71-15 
det command, Matlab, 71-17 

det function, Matlab, 71-3 

eig command, Matlab, 71-9, 71-17, 71-18 
eigenvalues, 15-5 

eigenvalues and eigenvectors, 71-9 
eigs function, 76—9 to 76-10 
eigshow command, Matlab, 71-9 
eigtool, Matlab, 16-12, 71-20 
ELMAT directory, Matlab, 71-5 

exp command, Matlab, 71-19 

expm function, Matlab, 11-11 
ezcontour command, Matlab, 71-15 


ezplot command, Matlab, 71-14, 71-14 to 71-15, 


71-17 
ezsurf command, Matlab, 71-17 
fname command, Matlab, 71-10, 71-11 
for loops, Matlab, 71-11 
format short command, Matlab, 71-8 
functions, Matlab, 71-11 
fundamentals, 71-1 
gallery command, Matlab, 71-5 
givens function, Matlab, 42-9 
graphical user interfaces, 71-19 to 71-22 
graphics, 71-14 to 71-17 
graphics, Matlab software, 71-14 to 71-17 
hilb command, Matlab, 71-18 
if statements, Matlab, 71-11 
implicitly restarted Arnoldi method, 44-1 
imread command, Matlab, 71-15 
inline command, Matlab, 71-11 to 71-12 
inv command, Matlab, 71-17 
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inv function, Matlab, 71-3 
jordan command, Matlab, 71-9 
kron function, Matlab, 71-4 
Lanczos methods, 42—21 
least squares solutions, 71—7 to 71-9 
left divide operator, Matlab, 71-7 
length command, Matlab, 71-2, 71-9 
linear systems, 71-7 to 71-9 
logical operators, Matlab, 71-12 
matrices, 71-1 to 71-3 
matrix arithmetic, 71—3 to 71-4 
max(size) command, Matlab, 71-2 
mesh command, Matlab, 71-15 
meshgrid command, Matlab, 71-14 
M-files, Matlab, 71-11, 71-20 
multidimensional arrays, 71-1 to 71-3 
narin command, Matlab, 71-12 
narout command, Matlab, 71-12 
norm command, Matlab, 71-17 
null command, Matlab, 71-8, 71-17 
numnull command, Matlab, 71-13 
orth command, Matlab, 71-17 
peaks function, Matlab, 71-17 
pi function, Matlab, 71-4 
planerot function, Matlab, 42-9 
plot command, Matlab, 71-14 
programming, 71-11 to 71-14 
rand command, Matlab, 71-6 
rank command, Matlab, 71-17 
relational operators, Matlab, 71-12 
right divide operator, Matlab, 71-7 
roots function, Matlab, 72-16 
rref command, Matlab, 71-17 
size command, Matlab, 71-2 
spare matrices, 71-9 to 71-11 
SPARFUN directory, Matlab, 71-10 
special matrices, 71-5 to 71-7 
spy command, Matlab, 71-10 
submatrices, 71-1 to 71-3 
subs command, Matlab, 71-17, 71-18 
sum command, Matlab, 71-17 
surfc command, Matlab, 71-15 
symbolic mathematics, 71-17 to 71-19 
sym command, Matlab, 71-17 
syms command, Matlab, 71-17 
toeplitz function, Matlab, 71-6 
trace command, Matlab, 71-17 
vpa command, Matlab, 71-17, 71-19 
while loops, Matlab, 71-11 

Matlab software’s EIGS subroutine package, 76-9 to 

76-10 

Matrices 
balanced, 32—5, 32-12 
bipartite graphs, 30—4 to 30-7 
classes of (0,1)-matrices, 27—7 to 27-10 
defective, 4-6 
derogatory, 4-6 
digraphs, 29-3 to 29-4 
dimension theorem, 2-6 to 2-9 
D-optimal matrices, 32-5 to 32-9, 32-12 
elementary, 1-11 to 1-13 


Index 


empty matrix, 68-3 

equal, 1-3 

factoring, 72-9 to 72-11 

function behaviors, 16-8 to 16-11 

fundamentals, 1-3, 1-3 to 1-6 

group representations, 68-3 to 68-5 

inverses, 1-11 to 1-13 

inversion, 47-9 

linear transformations, 3—3 to 3-4 

logarithm, 11-11 

Lyapunov diagonal stability, 19-9 

Maple software, 72—4 to 72-7, 72-16 to 72-18, 72-19 

to 72-20 

mappings, 18-11 

Mathematica software, 73—5 to 73-9, 73-13 to 73-14 

Matlab software, 71-1 to 71-3, 71-5 to 71-7 

max-plus algebra, 25-1 

nonregular, 32-7 to 32-9 

nonsquare case, 32—5 

norms, 37—4 to 37-6 

norms, error analysis, 37—4 to 37-6 

null space, 2-6 to 2-9 

rank, 2-6 to 2-9 

representations, 28—7 to 28-9 

singular values and singular value inequalities, 17—3 

to 17-5, 17-13 to 17-14 

spectral theory, 7-5 to 7-9 

systems of linear equations, 1-9 
(+1)-matrices, 31-8 
Matrices, combinatorial theory 

classes of (0,1)-matrices, 27—7 to 27-10 

convex polytopes, 27-10 to 27-12 

doubly stochastic matrices, 27-10 to 27-12 

monotone class, 27—7 to 27-8 

square matrices, 27-3 to 27-6 

strong combinatorial invariants, 27-3 to 27-5 

structure and invariants, 27—1 to 27-3 


tournament matrices, 27—8 to 27-10 
weak combinatorial invariants, 27—5 to 27-6 
Matrices, completion problems 

completely positive matrices, 35-10 to 35-11 

copositive matrices, 35-11 to 35-12 

doubly nonnegative matrices, 35-10 to 35-11 

entry sign symmetric P-, Po,,- and Po-matrices, 35-19 
to 35-20 

entry weakly sign symmetric P-, Pp,)- and 
Po-matrices, 35-19 to 35-20 

Euclidean distance matrices, 35—9 to 35-10 

fundamentals, 35-1 to 35-8 

inverse M-matrices, 35-14 to 35-15 

M- and Mo- matrices, 35-12 to 35-13 

nonnegative P-, Po,,- and Po-matrices, 35-17 to 
35-18 

P-, Po,1- and Po-matrices, 35-15 to 35-17 

positive definite matrices, 35-8 to 35-9 

positive P-matrices, 35-17 to 35-18 

positive semidefinite matrices, 35-8 to 35-9 

strictly copositive matrices, 35-11 to 35-12 

Matrices, equalities and inequalities 
determinantal relations, 14-10 to 14-12 
eigenvalues, 14—1 to 14-5, 14-8 to 14-10 
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inversions, 14—15 to 14-17 
nullity, 14-12 to 14-15 
rank, 14-12 to 14-15 
singular values, 14-8 to 14-10 
spectrum localization, 14-5 to 14-8 
Matrices, fast multiplication 
advanced techniques, 47—7 to 47-9 
algorithms, 47-2 to 47-5 
applications, 47-9 to 47-10 
approximation algorithms, 47-6 to 47-7 
fundamentals, 47—1 to 47-2 
Matrices, functions 
computational methods, 11-9 to 11-12 
cosine, 11—7 to 11-8 
exponential, 11-5 to 11-6 
fundamentals, 11-1 
logarithm, 11-6 to 11-7 
sign function, 11-8 
sine, 11-7 to 11-8 
square root, 11—4 to 11-5 
theory, 11-1 to 11-4 
Matrices, perturbation theory 
eigenvalue problems, 15-1 to 15-6, 15-9 to 15-11, 
15-13 to 15-15 
polar decomposition, 15-7 to 15-9 
relative distances, 15-13 to 15-16 
singular value problems, 15-6 to 15-7, 15-12 to 
15-13, 15-15 to 15-16 
Matrices, sign-pattern 
allowing properties, 33-9 to 33-11 
complex sign patterns, 33-14 to 33-15 
eigenvalue characterizations, 33-9 to 33-11 
fundamentals, 33-1 to 33-3 
inertia, minimum rank, 33-11 to 33-12 
inverses, 33—12 to 33-14 
L-matrices, 33—5 to 33-7 
orthogonality, 33-16 to 33-17 
powers, 33-15 to 33-16 
ray patterns, 33-14 to 33-16 
sign-central patterns, 33-17 
sign nonsingularity, 33-3 to 33-5 
sign solvability, 33-5 to 33-7 
S-matrices, 33—5 to 33-7 
stability, 33-7 to 33-9 
Matrices, sparse 
analyzing fill, 40-10 to 40-13 
effect of reorderings, 40-14 to 40-18 
factorizations, 40-4 to 40-10 
fundamentals, 40—1 to 40-2 
modeling, 40-10 to 40-13 
reordering effect, 40-14 to 40-18 
sparse matrices, 40-2 to 40-4 
Matrices, special properties 
canonical forms, 6-1 to 6-14 
Hermitian, 8—1 to 8-5 
nonnegative matrices, 9-1 to 9-15, 9-20 to 9-23 
partitioned matrices, 10-1 to 10-9 
positive definite matrices, 8-6 to 8-12 
spectral theory, 7-5 to 7-9 
stochastic matrices, 9-15 to 9-17 
unitary similarity, 7-1 to 7-5 
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Matrices, stability and inertia 
additive D-stability, 19-7 to 19-8 
fundamentals, 19-1 to 19-2 
inertia, 19-2 to 19-3 
Lyapunov diagonal stability, 19-9 to 19-10 
multiplicative D-stability, 19-5 to 19-7 
stability, 19-3 to 19-5 
Matrices computations, large-scale 
Arnoldi process, 49-10 to 49-11 
dimension reduction, 49-14 to 49-15 
eigenvalue computations, 49-12 
fundamentals, 49-1 to 49-2 
Krylov subspaces, 49-5 to 49-6 
linear dynamical systems, 49-14 to 49-15 
linear systems, equations, 49-12 to 49-14 
nonsymmetric Lanczos process, 49-8 to 49-10 
sparse matrix factorizations, 49-2 to 49-5 
symmetric Lanczos process, 49-6 to 49-7 
Matrices generation, Maple software, 72—4 to 
72-5 
Matrices over , 31-8 to 31-9 
Matrices sign function, 11-12 
Matrix, Maple software, 72-1, 72-4 to 72-5, 
72-13 to 72-14, 72-15 
matrix, Maple software, 72-1 
Matrix algebra, 73—9 to 73-12 
Matrix approximation, 17-12 to 17-13 
Matrix arithmetic, 71-3 to 71-4 
Matrix condition number, 38-2 
MatrixConditionNumber, Mathematica software 
matrix algebra, 73-10 
singular values, 73-18 
Matrix cosine and sine, 11-11 
Matrix direct sum, 2—7 
Matrix equations, control theory, 57-10 to 57-11 
Matrix equivalence, 23-4 to 23-8 
MatrixExp, Mathematica software, 73-9, 73-11 
Matrix exponential, 11-10 
MatrixExponential, Maple software, 72-19 
Matrix factoring, 72-9 to 72-11 
MatrixForm, Mathematica software 
decomposition, 73-19 
matrices, 73-6, 73-7, 73-8 
matrix algebra, 73-10, 73-11 
vectors, 73-2, 73-4, 73-5 
MatrixFunction, Maple software, 72-19 to 
72-20 
Matrix function behaviors, 16-8 to 16-11 
Matrix games, 50-18 to 50-20 
Matrix groups 
BN structure, 67—4 to 67-5 
classical groups, 67-5 to 67-7 
fundamentals, 67—1 to 67-3 
linear groups, 67-3 to 67-5 
Matrix in real-Jordan canonical form, 6-7 
MatrixInverse, Maple software, 72-5 
Matrix inversion, 47-9 
Matrix logarithm, 11-11 
Matrix mappings, 18-11 
Matrix multiplication complexity, 47-8 
MatrixNorm, Mathematica software, 73-27 
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Matrix pencils 
eigenvalue problems, 15-9 
extensions, 16-12 
generalized eigenvalue problem, 43-2 
linear differential-algebraic equations, 55-7 
Matrix perturbation theory 
eigenvalue problems, 15-1 to 15-6, 15-9 to 15-11, 
15-13 to 15-15 
polar decomposition, 15-7 to 15-9 
relative distances, 15-13 to 15-16 
singular value problems, 15-6 to 15-7, 15-12 to 
15-13, 15-15 to 15-16 
Matrix polynomial extension, 16-13 
Matrix polynomial of degree, 9-8 
MatrixPower, Mathematica software 
matrices, 73-6, 73-8 
matrix algebra, 73-9, 73-10, 73-11, 73-12 
Matrix power asymptotics, 25-8 to 25-9 
Matrix products 
fundamentals, 1-4 
Maple software, 72-6 
numerical range, 18-8 to 18-9 
MatrixQ, Mathematica software, 73-6 
Matrix quadratic forms, 53-8 to 53-11 
MatrixRank, Mathematica software 
fundamentals, 73-27 
matrix algebra, 73-10, 73-12 
Matrix representations, 28-7 to 28-9 
Matrix sign function, 11-12 
Matrix square root, 11-4 to 11-5, 11-11 
Matrix stability and inertia 
additive D-stability, 19-7 to 19-8 
fundamentals, 19-1 to 19-2 
inertia, 19-2 to 19-3 
Lyapunov diagonal stability, 19-9 to 19-10 
multiplicative D-stability, 19-5 to 19-7 
stability, 19-3 to 19-5 
Matrix-tree theorem, 28-8 
Matrix-vector multiplication, 37-8 
Matrix-vector product, 1-4 
Matrix-Vector products, Maple software, 72-6 to 
72-7 
Max algebra, 9-23 
Max-cut problem, 51-1, 51-2, 51-10 
Maximal cycle mean, 25-4 to 25-6 
Maximal ideal, 23-2 
Maximal rank, 33-11 
Maximal sign nonsingularity, 33-3 
Maximize, Mathematica software, 73-23, 73-27 
Maximum absolute column sum norm, 37—4 
Maximum absolute row sum norm, 37—4 
Maximum distance separable (MDS), 61-5 
Maximum distance separable (MDS) convolutional 
codes, 61-12 
Maximum entropy method, 64-14, 64-15 
Maximum likelihood estimates, 53-4 
Maximum multiplicity, 34—4 to 34-6 
Maximum rank deficiency, 34—4 
Max-plus algebra 
asymptotics, matrix powers, 25-8 to 25-9 
eigenproblem, 25-6 to 25-8 
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fundamentals, 25-1 to 25-4 
linear independence and rank, 25-12 to 25-14 
linear inequalities and projections, 25-10 to 
25-12 
maximal cycle mean, 25—4 to 25-6 
permanent, 25-9 to 25-10 
permanents, 25—9 to 25-10 
Max-plus Collatz-Wielandt formulas, 25-4 to 25-5 
Max-plus Cramer’s formula, 25-13 
Max-plus diagonal scaling, 25-6 
Max-plus eigenproblem, 25-6 to 25-8 
Max-plus permanent, 25-9 to 25-10 
Max-plus polynomial function, 25-9 
Max-plus semiring, 25-1 
Maxplus toolbox, 25-6 
max(size) command, Matlab software, 71-2 
Maxwell’s equations, 59-2 
McDonald studies, 26-7 
McLaurin expansion, 23-9 
McMillan degree, 57-6 
MDS (maximum distance separable), 61-5 
MDS (maximum distance separable) convolutional 
codes, 61-12 
Mean 
multivariate normal inference, 53—4 
statistics and random variables, 52-2 
Mean square prediction error, 64—7 
Mean value, 52-2 
Mean vectors, 52-3 
Measured outputs, 57-14 
Measure of relative separation, 17-7 
Mehrmann, Volker, 55-1 to 55-16 
Meini, Beatrice, 54—1 to 54-14 
Meini iteration, 11-11 
Menagé number, 31-6 
Menger matrix 
Euclidean simplexes, 66-8, 66-9 
fundamentals, 66-12 
resistive electrical networks, 66-15 
Merging 
matrix power asymptotics, 25-8 
nonnegative IEPs, 20-8 
Mesh, Mathematica software, 73-27 
mesh command, Matlab software, 71-15 
meshgrid command, Matlab software, 71-14 
Message blocks of length, 61-1 
Metabolites, 60-10 
Method of Condensation, 4—4 
Metric dynamical systems, 56-12 
Metric multidimensional scaling, 53-13 to 53-14 
Metric net, simplexes, 66-10 
Metrics, P—4 
Meyer, Carl D., 63-1 to 63-14 
M-files, Matlab software, 71-11, 71-20 
MGS (modified Gram-Schmidt) process, 44-4 
Midpoint 
affine spaces, 65-2 
Euclidean point space, 66-2 
MIEPs (multiplicative IEPs), 20-10 
Mills, Mark, 2-1 to 2-12 
Minimal connection, 29-12 to 29-13 


Minimally chordal symmetric Hamiltonian, 35-15 
Minimally potentially stable, 33-7 
Minimal matrix norms, 37—4 
Minimal polynomials 
convergence in gap, 44-9 
eigenvalues and eigenvectors, 4-6 
Krylov subspaces, 49-6 
Minimal rank, 33-11 
Minimal realization, 57-6 
Minimal Residual (MINRES) algorithm 
convergence rates, 41-14 to 41-15 
Krylov space methods, 41-4, 41-6, 41-10 
Minimal sign-central matrices, 33-17 
Minimax theorem, 50-18 
Minimize, Mathematica software 
fundamentals, 73-27 
linear programming, 73-23, 73-24 
Minimum co-cover, 27—2 
Minimum cover, 27-2 
Minimum deficiency algorithm, 40-17 
Minimum entropy controller, 57-15 
Minimum-norm least squares solution, 39-1 
Minimum phase, 64—2 
Minimum rank, 34—4 to 34-6 
Minimum rank inertia, 33-11 to 33-12 
Minors, see also Principal minors 
determinants, 4—1 
graphs, 28-4 
Minors, Mathematica software, 73-10, 73-11 
Min-plus semiring, 25-1 
MINRES (Minimal Residual) algorithm 
convergence rates, 41-14 to 41-15 
Krylov space methods, 41-4, 41-6, 41-10 
Minus algebra, 69-2 
Mirsky theorem, 15-6 
Mixed strategies, matrix games, 50-18 
M-matrices 
fundamentals, 9-17 to 9-20 
matrix completion problems, 35-12 to 35-13 
stability, 19-4 
Mo-matrices, 35-12 to 35-13 
M-norm, 37-2 
Mobius function, 20-7 
Model matrix, 52-8 
Models, see also specific model 
fill, sparse matrix methods, 40-10 to 40-13 
full-rank, 52-8 
Gauss-Markov model, 52-8, 53-11 
LAPACK subroutine package, 75-8 to 75-9 
linear statistical, 52-1 to 52-15 
multivariate linear model, 53-11 
multivariate statistical analysis, 53-11 to 53-13 
Padé, 49-14, 49-15 
Padé model, 49-14 
reduced-order model, 49-14 
signal model, 64-16 
univariate linear model, 53-11 
Modified Gram-Schmidt (MGS) process, 44—4 


Modified incomplete Cholesky decomposition, 41— 


Modular arithmetic, 72-12 to 72-13 
Module, Mathematica software, 73-26 
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Modules 
group representations, 68-2 
Lie algebras, 70-7 to 70-10 
matrix representations, 68-4 
Molecular distance geometry problem, 60-2 
Moment generating function, 53-3 
Monic polynomials, 23-2 
Monotone class, 27—7 to 27-8 
Monotone vector norm, 37-2 
Mood studies, 32-1 
Moore-Penrose inverse 
inverse patterns, 33-12, 33-13 
least squares solutions, 39-2 
linear statistical models, 52-12 
Maple software, 72-7 
Moore-Penrose pseudo-inverse 
extensions, 16-13 
pseudo-inverse, 5—12 
Morgan studies, 44-6 
Morse decompositions 
dynamical systems, 56—7 to 56-9 
Grassmannian and flag manifolds, 56-9 to 56-10 
robust linear systems, 56-17 
Morse sets, 56—7 
Morse spectrum, 56-16 
Most, Mathematica software 
fundamentals, 73-27 
matrices manipulation, 73-13 
vectors, 73-3 
Motzkin and Taussky studies, 7-8 
Moufang identities, 69-10 
Moulton Plane, 65-9 
(MRRR) multiple relatively robust representations, 
42-15 to 42-17 
Mukhopadhyay, Kriti, 60-13 
Multi-bigraph, 30-4 
Multidimensional arrays, 71-1 to 71-3 
Multigrid method, 41-3, 41-11 
Multilinear algebra 
alt multiplication, 13-17 to 13-19 
antisymmetric maps, 13-10 to 13-12 
associated maps, 13-19 to 13-20 
decomposable tensors, 13-7 
Grassmann tensors, 13-12 to 13-17 
Hodge star operator, 13-24 to 13-26 
inner product spaces, 13-22 to 13-24 
linear maps, 13-8 to 13-10 
multilinear maps, 13-1 to 13-3 
orientation, 13—24 to 13-26 
symmetric maps, 13-10 to 13-12 
symmetric tensors, 13-12 to 13-17 
sym multiplication, 13-17 to 13-19 
tensor algebras, 13-20 to 13-22 
tensor multiplication, 13-17 to 13-19 
tensor products, 13-3 to 13-7, 13-8 to 13-10, 13-22 
to 13-24 
Multilinear maps, 13-1 to 13-3 
Multinomial distribution, 52-4 
Multiple linear regression, 52-8 
Multiple relatively robust representations (MRRR), 
42-15 to 42-17 
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Multiplication, 69-1, see also Fast matrix multiplication 
Multiplication algebra, 69-5 
Multiplicative D-stability, 19-5 to 19-7 
Multiplicative Ergodic theorem, 56-14 to 56-15 
Multiplicative IEPs (MIEPs), 20-10 
Multiplicative perturbation 
eigenvalue problems, 15-13 
polar decomposition, 15-8 
singular value problems, 15-15 
Multiplicative preservers, 22—7 to 22-8 
Multiplicity 
algebraic connectivity, 36-10 to 36-11 
characters, 68-6 
composition, 13-13 
max-plus permanent, 25-9 
singular value decomposition, 45-1 
Multiplicity lists, see also Symmetric matrices 
double generalized stars, 34—11 to 34-14 
eigenvalues, 34—7 to 34-8 
fundamentals, 34—1 to 34—2 
generalized stars, 34-10 to 34-11 
maximum multiplicity, 34-4 to 34-6 
minimum rank, 34—4 to 34-6 
parter vertices, 34—2 to 34-4 
stars, generalized, 34-10 to 34-14 
trees, 34-8 to 34-10 
vines, 34-15 
Multiset, P—4 to P-5 
Multivariate Gauss-Markov theorem, 53-12 
Multivariate linear model, 53-11 
Multivariate normal distribution 
multivariate statistical analysis, 53-3 to 53-5 
positive definite matrices, 8-9 
Multivariate statistical analysis 
canonical correlations and variates, 53—7 
correlations and variates, 53-7 to 53-8 
data matrix, 53-2 to 53-3 
discriminant coordinates, 53-6 
estimation, correlations and variates, 53—7 to 53-8 
fundamentals, 53-1 to 53-2 
inference, multivariate normal, 53—4 to 53-5 
least squares estimation, 53-11 to 53-12 
matrix quadratic forms, 53-8 to 53-11 
metric multidimensional scaling, 53-13 to 53-14 
models, 53-11 to 53-13 
multivariate normal distribution, 53-3 to 53-5 
principal component analysis, 53-5 to 53-6 
statistical inference, 53-12 to 53-13 
Murakami, Lticia I., 69-1 to 69-25 
MUSIC algorithm, 64-17 
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Nagy, Kamm and, studies, 48-9 

Naming conventions, 76-3 to 76—4 

narin command, Matlab software, 71-12 

narout command, Matlab software, 71-12 

Narrow-band signals, 64-16 

Narrow sense Bose-Chadhuri-Hocquenghem (BCH) 
code, 61-8 
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Natural norms, 37—4 
Natural ordering, 41-2 
N-cycle matrices, 48-2 
Near breakdown, 49-8 
Nearby floating point numbers, 37-13 
Nearest-neighbor decoding, 61-2 
Nearly decomposable, 27-3 
Nearly reducible matrices, 29-12 to 29-13 
Nearly sign-central matrices, 33-17 
Nearly sign nonsingularity, 33-3 
Negative definite properties, 12-3, 12-8 
Negative half-life, 13-24 
Negative orientation, 13-24 
Negative semidefinite properties 
Hermitian forms, 12-8 
symmetric bilinear forms, 12-3 
Negative semistability, 33-7 
Negative stability 
sign pattern matrices, 33-7 
stability, 19-3 
Negative subdivision, 20-8 
Negative vertices, 36-7 
Neighbors, graphs, 28-2 
Nested basis, 49-5 
Nested dissection ordering, 40-16 to 40-17 
Net trace, 20—7 
Neubauer, Michael G., 32—1 to 32-12 
Neumann, Michael, 5—1 to 5-16 
Neumann boundary conditions, 59-10 
Neumann series, 14—16 
Newton iteration, 11-11 
Newton-Schultz iteration, 11-12 
Newton’s Law, 59-1 
Newton’s method 
interior point methods, 50-24 
numerical methods, PIEPs, 20-11 
primal-dual interior point algorithm, 51-8 
total least squares problem, 48-9 
Ng, Esmond G., 40-1 to 40-18 
Ng, Michael, 48-1 to 48-9 
Nielsen, Hans Bruun, 39—1 to 39-12 
Nil algebra, 69-5 
Nil ideal properties, 69-5 
Nilpotence 
alternative algebras, 69-10 
general properties, 69-5 
idempotence, 2-12 
invariant subspaces, 3—6 
reducible matrices, 9-11 to 9-12 
Nilpotency index, 69-5 
Nilpotent radical algebras, 69-6 
Nil radical algebras, 69-5 
Nil-semisimple algebras, 69-5 
Nodes, digraphs, 29-1 
Noise subspace, 64-16 
Noisy channel, 61-2 
Noisy transmission, 61-2 
Nonassociative algebra, 69-3, see also Algebra 
applications 
Akivis algebra, 69-16 to 69-17 
alternative algebras, 69-10 to 69-12 


composition algebras, 69-8 to 69-10 
computational methods, 69-20 to 69-25 
fundamentals, 69-1 to 69-4 

Jordan algebras, 69-12 to 69-14 

Malcev algebras, 69-16 to 69-17 
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noncommutative Jordan algebras, 69-14 to 69-16 


power associative algebras, 69-14 to 69-16 
properties, 69—4 to 69-8 
right alternative algebras, 69-14 to 69-16 
Sabinin algebra, 69-16 to 69-17 
Nonbasic variables, 50-10 
Noncentral distribution, 53-3 
Noncentral F-distribution, 53-3 
Noncollinear points, 65-2 
Noncommutative algorithms, 47-2 
Noncommutative Jordan algebra, 69-14 


Nondefective matrices, 4-6 
Nondegenerate properties 
bilinear forms, 12—2 
sesquilinear forms, 12-6 
Nonderogatory matrices, 4-6 
Nondifferentiation, 18—3 
Nonempty sets, row and column indices, 1—4 
Non-Hermitian case, 49-12 
Non-Hermitian Lanczos algorithm, 41-7 
Non-Hermitian problems, 41-7 to 41-11 
Nonhomogenous products, 9-22 
Nonlinear preservers, 22—7 to 22-8 
Nonnegative IEPs (NIEPs) 

fundamentals, 20—5 

merging results, 20-8 

nonzero spectra, 20—7 to 20-8 

spectra, 20-6 to 20-7 

sufficient conditions, 20-8 to 20-10 
Nonnegatives 

constraints, 50-3 

factorization, 9-22 

fundamentals, 9-1 

integer rank, 30-8 

matrix factorization, 63—5 to 63-8 

sign pattern matrices, 33-12 

stable matrices, 9-17 

vectors, 26-2 
Nonnegatives, matrices 
fundamentals, 9-1 to 9-2 
inequalities, 17-11 
inverse eigenvalue problem, 9-22 
irreducible matrices, 9-2 to 9-7 
max algebra, 9-23 
nonhomogenous products, 9-22 
nonnegative factorization, 9-22 


P-, Po,,- and Po-matrices, completion problems, 


35-17 to 35-18 
permanents, 31—7 
Perron-Frobenius theorem, 26—2 
product form, 9-23 
reducible matrices, 9-7 to 9-15 
scaling, 9-20 to 9-23 
sets, 9-23 

Nonnormality constant, 44-10 


Noncommutative Jordan algebras, 69-14 to 69-16 
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Non-optimal Krylov space methods, 41-7 to 41-11 
Nonprimary matrix function, 11-2 
Nonrandom matrices, 52-3 
Nonrandom vectors, 52-3 
Nonregular matrices, 32-7 to 32-9 
Nonscalar multiplications 

approximation algorithms, 47-6 

fast algorithms, 47—2 
Nonseparability, 35-2 
Nonsingular properties 

distribution, 53-8 

fundamentals, 2—9 to 2-10 

isomorphism, 3-7 to 3-8 

matrices, 1-12 

multivariate normal distribution, 53-3 
Nonsquare case, 32-2 to 32-12 
Nonsymmetric eigenproblems, 75-17 to 75-20 
Nonsymmetric eigenvalue problems, 75-11 to 

75-13 

Nonsymmetric Lanczos process 

Arnoldi process, 49-10 

large-scale matrix computations, 49-8 to 49-10 

linear dynamical systems, 49-15 
Nonsymmetric problems, ARPACK, 76-8 
Nonzero spectra, 20-7 to 20-8 
Norm, Maple software, 72-3, 72-5 
Norm, Mathematica software 

fundamentals, 73-26, 73-27 

matrix algebra, 73-10, 73-11 

vectors, 73-3, 73-5 
Normal, Mathematica software, 73-6, 73-8 
Normal equations 

least squares problems, 5-14 

linear statistical models, 52-8 
Normalization, 23-5 
Normalized properties 

floating point numbers, 37-11 

immanant, 31-13 

matrices, 25-6 

scaling nonnegative matrices, 9-20 
Normal vector, Euclidean point space, 66-2 
norm command, Matlab software, 71-17 
Norm estimation, 18—9 to 18-10 
Norms, matrices, 37—4 to 37-6 
Notation index, N—1 to N-9 
Notebooks, Mathematica software, 73-1 
Not invertible, 1-12, see also Invertibility 
Nth-derived algebra, 70-3 
null command, Matlab software, 71-8, 71-17 
Null graphs, 28-2 
Nullity 

linear independence, span, and bases, 2-6 

matrix equalities and inequalities, 14-12 to 

14-15 

Null recurrent state, 54—7 to 54-9 
NullSpace, Maple software 

matrix factoring, 72-9 

modular arithmetic, 72-14 to 72-15 
NullSpace, Mathematica software 

fundamentals, 73-27 

matrix algebra, 73-10, 73-12 
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Nullspaces, 39-4 
Null spaces 
dimension theorem, 2—6 to 2—9 
kernel and range, 3-5 
linear independence, span, and bases, 2-6 
matrix range, 2-6 to 2-9 
rank, 2-6 to 2-9 
Numerical linear algebra 
Maple software, 72-13 to 72-15 
support routines, 77-1 
Numerically orthogonal matrices, 46-2 
Numerical methods 
affine parameterized IEPs, 20-11 to 20-12 
fast matrix multiplication, 47-1 to 47-10 
high relative accuracy computation, 46-1 to 46-16 
implicitly restarted Arnoldi method, 44-1 to 44-12 
iterative solution methods, 41-1 to 41-17 
large-scale matrix computations, 49-1 to 49-15 
Markov chains, 54-12 to 54-14 
singular value decomposition, 45-1 to 45-12 
stability and instability, 37-18 to 37-21 
structured matrix computations, 48-1 to 48-9 
symmetric matrix techniques, 42—1 to 42-22 
unsymmetric matrix techniques, 43—1 to 43-11 
Numerical methods, linear systems 
direct solutions, 38—1 to 38-17 
efficiency, 37-1 to 37-21 
error analysis, 37-1 to 37-21 
factorizations, 38—1 to 38-17 
least squares solutions, 39-1 to 39-12 
matrix norms, 37-1 to 37-21 
sparse matrix methods, 40-1 to 40-18 
stability, 37-1 to 37-21 
vector norms, 37—1 to 37-21 
Numerical orthogonality, 46-2 
Numerical radius, 18-1 
Numerical range 
boundary points, 18-3 to 18-4 
dilations, 18—9 to 18-10 
examples, 18-1 to 18-3 
fundamentals, 18-1 
location, 18—4 to 18-6 
matrix mappings, 18-11 
matrix products, 18-8 to 18-9 
norm estimation, 18-9 to 18-10 
properties, 18—1 to 18-3 
radius, 18-6 to 18-8 
special boundary points, 18-3 to 18—4 
spectrum, 18-3 to 18-4 
unitary similarity, 7-2 
Numerical rank, 39-11 
Numerical stability and instability 
error analysis, 37-18 to 37-21 
Strassen’s algorithm, 47—4 
numnull command, Matlab software, 71-13 


O 


O and o, P-5 
Objective function, 50-1 
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Oblique Petrov-Galerkin projection 
eigenvalue computations, 49-12 
large-scale matrix computations, 49-2 

Observability Hessenberg form, 57-9 

Observability Kalman decomposition, 57—7 

Observability matrix, 57-7 

Observableness, control theory, 57-2 

Observer equation, 57-2 

Octonions, generalized, 69-4 

Odd cycle, 33-2 

Oettli-Prager theorem, 38-3 

Off-diagonal entry, 1-4 

Off-norm, Jacobi method, 42-17 

One-bit quantum gate, 62-2 

One(1)-chordal graphs, 35-2 

One-dimensional harmonic oscillator, 59-8 

One(1)-norm, 37-2 

One-sided Jacobi SVD algorithm 
high relative accuracy, 46—2 to 46-5 
positive definite matrices, 46-11 
preconditioned Jacobi SVD algorithm, 46-6 
singular value decomposition, 45-5 
symmetric indefinite matrices, 46-15 

One-to-one, kernel and range, 3-5 

Onto, kernel and range, 3-5 

Open halfspaces, 66-2 

Open sector, 33-14 

Operations and functions, Maple software, 72-3, 

72-5 

Operator norms 
matrix norms, 37-4 
unitary similarity, 7-2 

Optimal control problem, 57-14 

Optimal estimation problem, 57-12 

Optimality conditions, 51-5 to 51-7 

Optimality theorem, 51-6 

Optimal Krylov space methods, 41—4 to 41-11 

Optimal pivoting strategy, 42-17 

Optimal solution, 50-1 

Optimal value, 50-1 

Optimization 
linear programming, 50-1, 50-1 to 50-24 
matrix games, 50-18 
semidefinite programming, 51-1 to 51-11 
standard row tableaux, 50-8 

Orbit 
linear dynamical systems, 56—5 
simultaneous similarity, 24-8 

Order 
control theory, 57-2 
graphs, 28-1 
reducible matrices, 26-9 

Order predictable signals, 64—7 

Order sequence, 58-8 

Ordinary least squares estimator, 52-8 

Ordinary least squares solution, 52-8 

Orientation, 13-24 to 13-26 

Orientation preservation, 56-5 

Oriented incidence matrix, 28-7 

orth command, Matlab software, 71-17 

Orthogonality relations, 68—6 to 68-8 
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Orthogonalization, 38-13 to 38-15 
Orthogonal Petrov-Galerkin projection 
Arnoldi process, 49-10 
large-scale matrix computations, 49-2 
symmetric Lanczos process, 49-7 
Orthogonal properties 
classical groups, 67-5 
complement, 5-3 
congruence, 25-10 
Euclidean point space, 66-2 
Euclidean spaces, 65—4 
fundamentals, 5-3 to 5-5 
general properties, 69-5 
least squares solutions, 39-5 to 39-6 
linear inequalities and projections, 25-10 
projection, 5-6 to 5-8 
rank revealing decomposition, 39-11 
sign-pattern matrices, 33-16 to 33-17 
symmetric bilinear forms, 12-3 
symmetric indefinite matrices, 46-14 
symmetric matrix eigenvalue techniques, 42-2 
unitary similarity, 7-1 
Oscillation modes, 59-2 to 59-5 
Oscillatory matrices, 21-2 
Oseledets theorem, 56-14 to 56-15 
Ostrowski theorem 
eigenvalue problems, 15-13 
spectrum localization, 14-6 to 14-7 
Outer, Mathematica software, 73-2, 73-3, 73-4 
Outer normal, Euclidean simplexes, 66-7 
Outerplanar graphs, 28-4 
OuterProductMatrix, Maple software, 72-3 
Outlets, 66-13 
Output 
algorithms and efficiency, 37-16 
LTI systems, 57-14 
Output feedback, 57-7, 57-13 
Output space, 57-2 
Output vector, 57-2 
Ovals of Cassini, 14-6 to 14—7 
Overall constraint length, 61-12 
Overflow, floating point numbers, 37-12 


P 


P-, Po,j- and Po-matrices 
completion problems, 35-15 to 35-17 
stability, 19-3 
Packed format, 74-2 
Padé approximate, 11-10 to 11-11, 11-12 
Padé iterations, 11-12 
Padé models 
dimension reduction, 49-14 
linear dynamical systems, 49-15 
PadRight, Mathematica software, 73-13 
Page, Larry, 54—4, 63-9, 63-10 
PageRank 
fundamentals, 63-10 
information retrieval, 63-10 to 63-14 
Markov chains, 54—4 to 54—5 
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vector, 63-11 
Web search, 63-9 
Page repository, 63-9 
Pairwise orthogonality, 7-5 
Paley-Wiener theorem, 64-3 to 64-4 
Pan, V., 47—7 
Pappus’ theorem, 65-8, 65-9 
Parabolic subgroup, 67—4 
Parabolic subgroup, BN structure, 67-5 
Parallelepiped, Gram matrices, 66-5 
Parallel hyperplanes, 66-2 
Parallelogram, Gram matrices, 66-5 
Parallelogram law, inner product spaces, 5-3 
Parallel vector subspace, 65-2 
Parameters, graphs, 28—9 to 28-11 
Parametric programming, 50-17 to 50-18 
Parametrization of correlation matrices, 8-8 
Parity check 
matrix, 61-3 
polynomial, 61-7 
Parlett and Reinsch studies, 43-3 
Parlett’s recurrence, 11-10, 11-11 
Parseval’s Inequality, 5-4 
Parter vertices, 34—2 to 34-4 
Parter-Wiener theorem 
given multiplicities, 34-9 
multiplicities and Parter vertices, 34-2 
Parter-Wiener vertex, 34-2 
Partial completely positive matrices, 35-10 
Partial copositive matrices, 35-11 
Partial correlation coefficient 
linear prediction, 64-8 
random vectors, 52—4 
Partial correlation matrix, 52-4 
Partial covariance matrix, 52—4 
Partial differential equations, 58-7 
Partial doubly nonnegative matrices, 35-10 
Partial entry sign symmetric P-, Po,,- and Po-matrices, 
35-19 to 35-20 
Partial entry weakly sign symmetric P-, Po,)- and 
Po-matrices, 35-19 to 35-20 
Partial Euclidean distance matrices, 35—10 
Partial inverse M-matrices, 35-14 
Partial matrices, 35-2 
Partial matrix multiplication, 47-8 
Partial M-matrices, 35-12 to 35-13 
Partial Mo-matrices, 35-12 to 35-13 
Partial nonnegative P-, Po,,- and Po-matrices, 35-17 to 
35-18 
Partial order, checkerboard, 21-9 
Partial P-, Po,1- and Po-matrices, 35-15 
Partial pivoting, 40-18, see also Pivoting 
Partial positive definite matrices, 35-8 
Partial positive P-matrices, 35-17 
Partial positive semidefinite matrices, 35-8 
Partial Schur decomposition, 44-6, 44-8 
Partial semidefinite ordering 
positive definite matrices, 8-10 
random vectors, 52—4 
Partial strictly copositive matrices, 35-11 
Partition, Mathematica software, 73-14 


Handbook of Linear Algebra 


Partitioned matrices 
block diagonal matrices, 10-4 to 10-6 
block matrices, 10—1 to 10-3 
block triangular matrices, 10-4 to 10-6 
Kronecker products, 10-8 to 10-9 
random vectors, 52—3 
Schur complements, 10-6 to 10-8 
submatrices, 10-1 to 10-3 
submatrices and block matrices, 10-1 
Partly decomposable, 27-3 
Pascal matrices 
factorizations, 21-6 
totally positive and negative matrices, 21-4 
Passage class, 54-5 
Path 
digraphs, 29-2 
modeling and analyzing fill, 40-10 
sign-pattern matrices, 33-2 
Path-connection, P—5 
Path cover number, 34—4 
Path of length, 28-1, 28-2 
Pattern block triangular form, 35-2 
Patterns, 27-1, see also Sign-pattern matrices 
Payoff matrix, 50-18 
Peak characteristics, 26-9 
peaks function, Matlab software, 71-17 
PEIEPs (prescribed entries inverse eigenvalue 
problems), 20-1 to 20-3 
PEIPs, see Affine parameterized IEPs (PIEPs) 
Pencils, matrix 
generalized eigenvalue problem, 43-2 
linear differential-algebraic equations, 55-7 
matrices over integral domains, 23-9 to 23-10 
Pendant path, 34-2 
Penrose conditions, 5—12 
Perfect code, 61-2 
Perfect elimination ordering 
bipartite graphs, 30-4 
reordering effect, 40-14 
Perfectly well determined, high relative accuracy, 
46-8 
Perfect matching, 30-2 
Period 
complex sign and ray patterns, 33-14 
Fourier analysis, 58-2 
imprimitive matrices, 29-10 
irreducible matrices, 9-3 
linear dynamical systems, 56-5 
reducible matrices, 9—7 
Simon’s problem, 62-13 
Periodic characteristics 
Fourier analysis, 58-2 
irreducible classes, 54—5 
linear dynamical systems, 56-5 
Periodic linear differential equations, 56-12 to 
56-14 
Peripheral eigenvalues, 26-6 
Peripheral spectrum 
cone invariant departure, matrices, 26—5 to 
26-7 
Perron-Schaefer condition, 26-6 
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perm, Mathematica software, 73-19 
permanent, Mathematica software, 73-12 
Permanental dominance conjecture, 31-13 
Permanents 
binary matrices, 31-5 to 31-7 
determinant connections, 31—12 to 31-13 
doubly stochastic matrices, 31-3 to 31-4 
evaluation, 31—11 to 31-12 
fundamentals, 31-1 to 31-3 
(+1)-matrices, 31-8 
matrices over , 31-8 to 31-9 
max-plus algebra, 25-9 to 25-10 
nonnegative matrices, 31-7 
rook polynomials, 31-10 to 31-11 
subpermanents, 31-9 to 31-10 
permMatr, Mathematica software, 73-19 
Permutation invariants 
absolute norm, 17-5 
vector norms, 37—2 
Permutation matrix 
fundamentals, 1-6 
Gauss elimination, 38-7 
matrices, 1-4 
systems of linear equations, 1-13 
Permutations 
eigenvalue problems, 15-2 
equivalence, 31-1 
fundamentals, P—5 
pattern, 33-2 
representation, 68-3 
restricted positions, 31-6 
similarity, 27-5, 33-2 
Perpendicular bisectors, 65-4 
Perpendicular hyperplanes, 66-2 
Perron branch, 36-5 
Perron component, 36-10 
Perron-Frobenius theorem 
cone invariant departure, matrices, 26-1 to 
26-3 
elementary analytic results, 26-12 
irreducible matrices, 9—4, 29-6 
Perron-Frobenius-type results, 9-10 
Perron-Schaefer condition, 26—5 to 26-7 
Perron spaces 
Collatz-Wielandt sets, 26—4 
K-reducible matrices, 26-10 
Perron’s theorem, 9-3 
Perron value 
convergence properties, 9-9 to 9-10 
irreducible matrices, 9—4 to 9-5 
nonnegative and stochastic matrices, 9-2 
nonzero spectra, 20—7 
reducible matrices, 9-8 to 9-10 
Persistently exciting of order, 64-12 
Personalization vector, 63-11 
Pertinent pages, Web search, 63-9 
Perturbations, 38-2 to 38—5 
Perturbation theory, matrices 
eigenvalue problems, 15-1 to 15-6, 15-9 to 
15-11, 15-13 to 15-15 
polar decomposition, 15-7 to 15-9 


relative distances, 15-13 to 15-16 


singular value problems, 15-6 to 15-7, 15-12 to 


15-13, 15-15 to 15-16 
Perturbed linear system, 38-2 
Petersen graphs 
adjacency matrices, 28-7 
embedding, 28-5 
graph parameters, 28-10 
Laplacian, 28-9 
nonsquare case, 32—11 
Petrie matrix, 30—4, 30-5, 30-7 
Petrov-Galerkin projection 
Arnoldi process, 49-10 
eigenvalue computations, 49-12 
Krylov subspaces, 49-5 
large-scale matrix computations, 49-2 
symmetric Lanczos process, 49-7 
Pfaffian properties, 12-5 
Phase 2 geometric interpretation, 50-13 
Phillips regularization, 39-9 
Physical sciences applications, 59-1 to 59-11 
Pick’s inequalities, 14-2 
PID (principal ideal domain), 23-2 
Pierce decomposition, 69-13 
Pi function, Mathematica software, 73-26 
pi function, Matlab software, 71—4 
Pillai’s trace statistic, 53-13 
Pinching, inequalities, 17—7 
Pisarenko’s method, 64—14 
Pi theorem, 47-8, 47-9 
Pivot column, 1-7 
Pivoted QR factorization, 39-11 
Pivoting 
elements, Jacobi method, 42-17 
Gauss elimination, 1-7, 38—7 
Gauss-Jordan elimination, 1—7 
Jacobi method, 42-17, 42-18 
linear programming, 50-10 to 50-11 
LU decomposition, 38-10 
positions, 1-7 
preconditioned Jacobi SVD algorithm, 46-5 
reordering effect, 40-18 
Pivot row, 1-7 
Pivot step, 50-10 
Planar graphs, 28-4 
Planck’s constant, 59-6 
Plane, affine spaces, 65-2 
planerot function, Matlab software, 42-9 
PLDU factorization, 1-14 to 1-15 
p-Lie algebra, 70-2 
plot command, Matlab software, 71-14 
Plotkin bound, 61-5 
PLU factorization, 1-13 
Plus, Mathematica software, 73-27 
Plus algebra, 69-3 
p-norm, 37-2 
Poincaré-Bendixson theorem, 56-8 
Poincaré-Birkhoff-Witt theorem 
Lie algebras, 70-3 
Malcev algebras, 69-17 
nonassociative algebra, 69-3 
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Pointed cones, 8-10 generalized eigenvectors, 9-11 
Points half-life, 13-24 
affine spaces, 65-2 matrix mapping, 18-11 
Euclidean point space, 66-1 orientation, 13-24 
projective spaces, 65-6 Perron-Frobenius theorem, 26-2 
Point spaces, 66—1 to 66-5 P-matrices, 35-17 to 35-18 
Point-wise bounded, 37-3 recurrent state, 54—7 to 54-9 
Point-wise similarity, 24—1 root, semisimple and simple algebras, 70—4 
Poisson’s equation stability, 19-3, 26-13 
discrete theory, 58-12 stable matrices, 9-17 
Green’s functions, 59-11 vector seminorms, 37-3 
Polar cones, 51-3 Positive semidefinite properties 
Polar decomposition completion problems, 35-8 to 35-9 
perturbation theory, 15-7 to 15-9 Hermitian forms, 12-8 
singular values, 17-1 matrix completion problems, 35-8 
PolarDecomposition, Mathematica software, 73-19 positive definite matrices, 8-6 
Polar form semidefinite programming, 51-3 
positive definite matrices, 8-7 symmetric bilinear forms, 12-3 
singular values, 17-1 Potential energy 
Polarization formula, 12-8 Lagrangian mechanics, 59-5 
Polarization identity, 5-3, 13-11 oscillation modes, 59-2 
Policy iteration algorithm, 25-7 protein motion modes, 60-9 
Polya matrix, 21-11 Potentially orthogonality (PO), 33-16 
Polyhedral cones, 26-4 Potential matrix, 54~—7 to 54-9 
Polynomials Potentials, duality, 50-17 
adjacency matrix, 28-5 Potential stability, 33-7 
certain integral domains, 23-1 Potent square sign or ray patterns, 33-14 
eigenvalues and eigenvectors, 4-6, 4-8 Powell-Reid’s complete pivoting, 46-5, 46-7 
interpolation, 11-2 Power 
linear code classes, 61—7 algorithm, 25-7 
numerical hull, 41-16 convergence properties, 9-4, 9-9 
restarting, 44-5 to 44-6 irreducible matrices, 9—4 
span and linear independence, 2-2 matrices, 1-12 
stability, 19-3 method, 42-2 
vector spaces, 1-3 mth, matrices, 1-12 
zeros, 37-8 reducible matrices, 9-9 
PolynomialTools, Maple software, 72-20, 72-21 sign-pattern matrices, 33-15 to 33-16 
PO (potentially orthogonality), 33-16 symmetric and Grassmann tensors, 13-12 
Population, 52-2 tensor products, 13-3 
Population canonical correlations and variates, Power associative algebras 
53-7 general properties, 69-5 
Population principal component, 53-5 nonassociative algebra, 69-14 to 69-16 
Positionally symmetric partial matrix, 35-2 Precision, Mathematica software, 73-17 
Positive definite Jacobi EVD, 46-11 Precisions 
Positive definite matrices (PSD), see also ARPACK subroutine package, 76-3 to 76-4 
Completely positive matrices floating point numbers, 37-12 
completion problems, 35-8 to 35-9 vector space method, 63-2 
fundamentals, 8-6 to 8-12 Preconditioned conjugate gradient (PCG) algorithm, 
high relative accuracy, 46-10 to 46-14 41-13 
inner product spaces, 5-2 Preconditioned Jacobi SVD algorithm, 46-5 to 46-7 
symmetric factorizations, 38-15 Preconditioners, iterative solution methods 
Positive definite properties algorithms, 41-12 to 41-14 
Hermitian forms, 12-8 fundamentals, 41-11 to 41-12 
matrix norms, 37—4 Krylov subspaces, 41-2 to 41-4 
symmetric bilinear forms, 12-3 Preconditioning 
vector norms, 37—2 algorithms, 41-12 to 41-14 
vector seminorms, 37-3 coefficient matrix, 49-4 
Positive/null recurrent state, 54—7 to 54-9 iteration matrices, 41-3 
Positive properties, see also Nonnegatives Jacobi SVD algorithm, 46-5 to 46-7 
eigenvectors, 9-11 Krylov subspaces and preconditioners, 41-3 


Fiedler vectors, 36—7 sparse matrix factorizations, 49-4 
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Predicable random signals, 64—7 
Prediction error, 64—7 
Prediction error filter, 64—7 
Prefixes, 74—2 
Prepend, Mathematica software 
matrices manipulation, 73-13 
vectors, 73-3 
Prescribed entries inverse eigenvalue problems 
(PEIEPs), 20-1 to 20-3 
Prescribed-line-sum scalings, 9-21 
Preservation, see also Linear preserver problems 
bilinear forms, 12-2 
linear dynamical systems, 56-5 
Lyapunov diagonal stability, 19-9 
multiplicative D-stability, 19-5 
sesquilinear forms, 12-6 
Primal-dual interior point algorithm, 51-8 to 
51-9 
Primary decomposition, 6-9 
Primary Decomposition Theorem 
eigenvectors, 6—2 
rational canonical form, 6-10 
Primary factors, 6-9 
Primary matrix function, 11-2 
Prime elements, 23-2 
Primitive Bose-Chadhuri-Hocquenghem (BCH) 
code, 61-8 
Primitive properties 
certain integral domains, 23-2 
digraphs and matrices, 29-8 to 29-9 
elements, 69-18 
gates, 62-7 
Principal angles, 17-14, 17-15 
Principal Axes Theorem, 7—5 to 7-6 
Principal character, 68-5 
Principal component, 26-12 
Principal component analysis, 53-5 to 53-6 
Principal eigenprojection, 26-12 
Principal ideal domain (PID), 23-2 
Principal ideals, 23-2 
Principal logarithm, 11-6 
Principal minors 
determinants, 4—3 
stability, 19-3 
Principal parts, 24-1 
Principal square root, 11-5 
Principal submatrix 
fundamentals, 1-6 
matrices, 1—4 
reducible matrices, 9-7 
submatrices and block matrices, 10-1 
Principal vectors, 17-14 
Probability, 52-2 
Probability and statistics applications 
linear statistical models, 52-1 to 52-15 
Markov chains, 54—1 to 54-14 
multivariate statistical analysis, 53-1 to 53-14 
random vectors, 52—1 to 52-15 
Probability density function, 52-2 
Probability function, 52-2 
Probability vector, 4-10 
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Procrustes problem, 60—4 to 60-7 
Product 
algebraic connectivity, 36-1 
characters, 68—5 
vector spaces, 3-2 
Product form, 9-23 
Product-moment correlation, 52-3 
Profile methods, 40-10, 40-16 
Profiles, reordering effect, 40-14 
Programming 
associated linear programming, 50-14 
Delsarte’s Linear Programming Bound, 28-12 
dynamic, 25-3 
LinearProgramming, Mathematica software, 
73-24 
linear semidefinite, 51-3 
mathematical, 50-1 
parametric, 50-17 to 50-18 
programming, Matlab software, 71-11 to 71-14 
symmetric cone, 51-2 
Programming, linear 
canonical forms, 50-7 to 50-8 
duality, 50-13 to 50-17 
formulation, 50-3 to 50-7 
fundamentals, 50-1 to 50-2 
geometric interpretation, phase 2, 50-13 
interior point methods, 50—23 to 50-24 
linear approximation, 50-20 to 50-23 
Mathematica software, 73—23 to 73-24 
matrix games, 50-18 to 50-20 
parametric programming, 50-17 to 50-18 
phase 2 geometric interpretation, 50-13 
pivoting, 50-10 to 50-11 
sensitivity analysis, 50-17 to 50-18 
simplex method, 50-11 to 50-13 
standard forms, 50-7 to 50-8 
standard row tableaux, 50-8 to 50-10 
Programming, Matlab software, 71-11 to 71-14 
Programming, semidefinite (SDP) 
applications, 51-9 to 51-11 
constraint qualification, 51-7 
duality, 51-5 to 51-7 
fundamentals, 51-1 to 51-3 
geometry, 51-5 
notation, 51-3 to 51-5 
optimality conditions, 51-5 to 51-7 
primal-dual interior point algorithm, 51-8 to 
51-9 
results, 51-3 to 51-5 
strong duality, 51-7 
Projection, Mathematica software, 73-4, 73-5 
Projectionally exposed face, 51-5 
Projection formulas, 44-2 
Projections, 3-6, 3-6 to 3-7 
Projective general linear group, 67-3 
Projective plane 
linear code classes, 61-10 
projective spaces, 65-6 
Projective spaces, 65-6 to 65-9 
Projective special linear group, 67-3 
Projective transformation, 65-7 
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Propagator, 59-10 
Proper cones, 26-1 
Proper digraphs, 29-2 
Properly signed nest, 33-7 
Proper point, Euclidean simplexes, 66-8 
Properties 
nonassociative algebra, 69-4 to 69-8 
numerical range, 18—1 to 18-3 
Property C, satisfying, 9-17 
Property L 
similarity of matrix families, 24-6 to 24-7 
spectral theory, 7-5 
Protein motion mode calculation, 60—9 to 60-10 
Proximity measure, 53-13 
PSD, see Positive definite matrices (PSD) 
Pseudo-code, 37-16 
Pseudoeigenvalues, 16-1 
Pseudoeigenvectors, 16-1 
Pseudo-inverse, 5—12 to 5-14 
PseudoInverse, Mathematica software 
linear systems, 73-20, 73-23 
matrix algebra, 73-10, 73-11 
Pseudospectra 
computation, 16-11 to 16-12 
extensions, 16—12 to 16-15 
fundamentals, 16-1 to 16-5 
matrix function behaviors, 16—8 to 16-11 
Toeplitz matrices, 16-5 to 16-8 
Pseudospectral abscissa 
matrix function behavior, 16-8 
Pseudospectral radius, 16-8 
Pseudospectrum 
convergence in gap, 44-10 
rectangular matrix, 16-12 
Puiseaux expansion 
matrix similarities, 24—1 
matrix similarity, 24-2 
Puiseux expansions, 9-10 
Puntanen, Simo, 52-1 to 52-15, 53-1 to 53-14 
Pure strategies, matrix games, 50-18 
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QFT (Quantum Fourier transform), 62-6 

QMR (quasi-minimal residual) algorithm 
Krylov space methods, 41-8, 41-10 to 41-11 
linear systems of equations, 49-13 
preconditioners, 41-12 

Q-norm, 17-6 

QR decomposition, 38-13 to 38-15 

QRDecomposition, Maple software, 72-9 

QRDecomposition, Mathematica software, 73-18 

QR factorization, see also Factorizations 
algorithm efficiency, 37-17 
Gram-Schmidt orthogonalization, 5-8 to 5-10 
least squares solutions, 39-8 to 39-9 
numerical stability and instability, 37-20 
orthogonal factorizations, 39-5 
preconditioned Jacobi SVD algorithm, 46-5 
rank revealing decomposition, 39-11 
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QR iteration 
explicit, 43-5 to 43-6 
symmetric matrix eigenvalue techniques, 42-3 
QR method, see Implicitly shifted QR method 
Quadrangular bipartite graph, 30-1 
Quadratic algebras, 69-8 
Quadratic forms 
fundamentals, 12-1, 12-3 to 12-5 
matrices, 53-8 to 53-11 
Qualitative class 
complex sign and ray patterns, 33-14 
sign-pattern matrices, 33-1 
Quantum bit, 62-2 
Quantum circuit, 62-2 
Quantum computation 
Bernstein-Vazirani problem, 62-11 to 62-13 
Deutsch-Jozsa problem, 62-9 to 62-11 
Deutsch’s problem, 62-8 to 62-9 
fundamentals, 62—1 to 62-7 
Grover’s search algorithm, 62-15 to 62-17 
Shor’s factorization algorithm, 62-17 to 62-19 
Simon’s problem, 62-13 to 62-15 
universal quantum gates, 62—7 to 62-8 
Quantum Fourier transform (QFT), 62-6 
Quantum register, 62-2 
Quantum Turing machine, 62-2 
Quarternions, generalized, 69-4 
Quartics, Mathematica software, 73-14 
Quasi-associative algebras, 69-15 
Quasi-irreducibility characteristics, 24-8, 24-11 
Quasi-minimal residual (QMR) algorithm 
Krylov space methods, 41-8 
linear systems of equations, 49-13 
preconditioners, 41-12 
Quasi-triangular characteristics, 43-6 
Query module, 63-9 
Query processing, 63-2 
Query vector, 63-2 
Queueing system, 54—4 
Quotient, direct sum decompositions, 2-5 
Quotient algebra, 69-4 
Quotient field, 23-1 
Quotient representation, 68-1 
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Radical algebras, 69-5, 70-4 
Radius, 18-6 to 18-8 
Radix 2 FFT, 58-17 to 58-19 
Raising operator, 59-8, 59-9 
rand command, Matlab software, 71-6 
Random linear dynamical systems, see also Dynamical 
systems 

fundamentals, 56-14 to 56-16 

linear skew product flows, 56-12 
Random samples, data matrix, 53-3 
Random signals, 64—4 to 64—7 
Random vectors 

fundamentals, 52-1 to 52-8 

linear statistical models, 52-8 to 52-15 
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Random walk, Markov chains, 54—3 to 54-4 
Range 
kernel, 3—5 to 3-6 
least squares solution, 39-4 
linear independence, span, and bases, 2-6 
linear inequalities and projections, 25-10 
Range, Mathematica software, 73-3, 73-4 
Rank 
bilinear forms, 12-2 
combinatorial matrix theory, 27-2 
convolutional codes, 61-11 
decomposable tensors, 13-7 
decompositions, bipartite graphs, 30-8 
dimension theorem, 2—6 to 2-9 
Gaussian and Gauss-Jordan elimination, 1—7 
inertia, 33-11 
kernel and range, 3-5 
linear independence, 2-6, 25-13 
matrix equalities and inequalities, 14—12 to 14-15 
matrix range, 2-6 to 2-9 
null space, 2-6 to 2-9 
semisimple and simple algebras, 70-4 
sesquilinear forms, 12-6 
Rank, Maple software, 72-9 
rank command, Matlab software, 71-17 
Rank-deficient least squares problem, 5-14 
Rank Equalities method, 2—7 to 2-8 
Rank Inequalities method, 2-7 to 2-8 
Ranking module, 63-9 
Rank revealing, 46-5 
Rank revealing decomposition (RRD) 
high relative accuracy, 46—7 to 46-10 
least squares solutions, 39-11 to 39-12 
Rank revealing QR (RRQR) decomposition, 39-11 
Rate, linear block codes, 61-3 
Rational canonical forms (RCF) 
elementary divisors, 6-8 to 6-11 
invariant forms, 6-12 to 6-14 
matrix similarity, 24-3, 24-4 
Rational similarity, 24-1 
Ravindrudu, Rahul, 60-13 
Rayleigh quotient 
Arnoldi factorization, 44—3 
Hermitian matrices, 8-3 
symmetric matrix eigenvalue techniques, 42-3 
total least squares problem, 48-9 
Rayleigh-Ritz inequalities, 14—4 
Rayleigh-Ritz theorem, 8-3, 8—4, 8-5 
Ray nonsingular pattern, 33-14 
Ray patterns, 33-14 to 33-16 
RCE, see Rational canonical forms (RCF) 
Reaction equations, 60-10 
Real affine space, 65-2 
Real division algebra, 69-4 
Realization, 57-6 
Real-Jordan block, 6-7 
Real-Jordan canonical form, 6—6 to 6-8, see also Jordan 
canonical form 
Real Jordan form, 56-2 
Real-Jordan matrix, 6—7 
Real square matrices, 19-5, 19-9 
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Real structured pseudospectrum, 16-12 
Reams, Robert, 10-1 to 10-9 
Recall, vector space method, 63-2 
Recognition 
matrix power asymptotics, 25-8 
total positive and total negative matrices, 21-6 to 21-7 
Reconstructibility, 57-2 
Rectangular matrix multiplication, 47-5 
Rectangular matrix pseudospectrum, 16-12 
Recurrent state, 54—7 to 54-9 
Recursive least squares (RLS), 64-12 
Reduce, Mathematica software, 73-20, 73-21 
Reduced digraphs 
irreducible matrices, 29-7 
nonnegative and stochastic matrices, 9-2 
reducible matrices, 9-7 
Reduced-order model, 49-14 
Reduced QR factorization, 5-8 
ReducedRowEchelonForm, Maple software, 72-9, 72-10 
Reduced row echelon form (RREF) 
computational methods, 69-23, 69-25 
Gaussian and Gauss-Jordan elimination, 1—7 to 1-9 
rank, 2-6 
systems of linear equations, 1-10 to 1-11, 1-12, 1-13 
Reduced singular value decomposition (reduced SVD) 
fundamentals, 45-1 
singular value decomposition, 5-10 to 5-11 
Reducibility 
group representations, 68-1 
matrix group, 67-1 
matrix representations, 68-3 
modules, 70—7 
square matrices, weak combinatorial invariants, 27-5 
Reducible matrices 
cone invariant departure, matrices, 26-8 to 26-10 
fundamentals, 9-7 to 9-15 
max-plus eigenproblem, 25-7 
nonnegative matrices, 9-7 to 9-15 
Reducing eigenvalue, 18-3 
Redundancy, 50-4 
Reed-Solomon code, 61-8, 61-9, 61-10 
REF, see Row echelon form (REF) 
Reflection, 70—4 
Reflection coefficients, 64—8 
Reflection matrix, 65-5 
Regression, random vectors, 52-4 
Regressor vectors, 52-8 
Regular bimodule algebras, 69-6 
Regular graphs, 28-3 
Regularly cyclic simplexes, 66-12 
Regular matrices, 32-5 to 32-7, see also Matrices 
Regular matrix pencils, 55-7 
Regular pencils, 43-2 
Regular point, 24-8 
Regular signals, 64—7 
Regular splitting 
Krylov subspaces and preconditioners, 41-3 
numerical methods, 54—12 
Regulated output, 57-14 
Reinsch, Parlett and, studies, 43-3 
Relational functions field, 23—2 
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Relational operators, Matlab software, 71-12 
Relative backward errors, linear system, 38-2 
Relative condition number, 37—7 
Relative distances, 15-13 
Relative errors 
conditioning and condition numbers, 37-7 
floating point numbers, 37-13, 37-16 
Relative perturbation theory 
eigenvalue problems, 15-13 to 15-15 
singular value problems, 15-15 to 15-16 
Relative separation measure, 17—7 
Relevance, vector space method, 63-2 
Reordering effect, 40-14 to 40-18 
Representation 
group representations, 68-2 to 68-3 
Malcev algebras, 69-16 
modules, 70—7 
Residual matrix, 52-9 
Residuals 
Krylov subspaces and preconditioners, 41-2 
least squares problems, 5-14 
linear approximation, 50-20 
linear statistical models, 52-8 
random vectors, 52—4 
Residual sum of squares, 52-8 
Residual vector 
least squares solution, 39-1 
linear system perturbations, 38-2 
Resistive electrical networks, 66-13 to 66-15 
Resolvents 
expansions, 9-10 
nonnegatives, 26-13 
pseudospectra, 16-1 
Respectively definite matrices, 51-3 
Rest, Mathematica software, 73—3, 73-13 
Restarted GMRES algorithm, 41-7 
Restarting process, 44—4 to 44-5 
Restricted subspace dimensions, 44-10 
Retrieved documents, 63—2 
Reverse, Mathematica software, 73-27 
Reverse communication, 76-2 
rhs, Mathematica software, 73-20 
Riccatti equation, 51-9 
Ridge aggression, 39-9 
Rigal-Gaches theorem, 38-3 
Right alternative algebras, 69-10, 69-14 to 69-16 
Right alternative identities, 69-2 
Right deflating subspaces, 55-7 
Right divide operator, Matlab software, 71-7 
Right Kronecker indices, 55-7 
Right Krylov subspace 
Arnoldi process, 49-10 
nonsymmetric Lanczos process, 49-8 
Right Lanczos vectors, 49-8 
Right-looking methods, 40-10 
Right Moufang identity, 69-10 
Right multiplication operators, 69-5 
Right nilpotency, 69-14 
Right preconditioning, 41-3 
Right reducing subspaces, 55-7 
Right simplexes, 66-10 
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Right singular space, 45-1 

Right singular vectors, 45-1 

Rigid motion, 65-4 

Ring, P-6 

Ring automorphism, 22-7 

Ritz pairs 
Arnoldi factorization, 44-3 
spare matrices, 43-10 

Ritz values 
implicit restarting, 44-8 
Krylov subspace projection, 44—2 
spare matrices, 43-10 

Ritz vectors 
Krylov subspace projection, 44-2 
polynomial restarting, 44-6 
spare matrices, 43-10 

RLS (recursive least squares), 64—12 


RMSD (root-mean-square deviation), 60-4 to 60-7 


Robinson, J., 50-24 
Robust linear systems 
dynamical systems, 56-16 to 56-19 
linear skew product flows, 56-12 
Robust representations, 42-15 to 42-17 
Romani, R., 47-8 
Rook numbers, 31—10 
Rook polynomials, 31-10 to 31-11 
Root, positive definite matrices, 8-6 


Root-mean-square deviation (RMSD), 60-4 to 60-7 


RootOf, Maple software, 72-11, 72-20 
roots function, Matlab software, 72-16 
Root space, 70—4 
Root system, 70—4 
Rosenthal, Joachim, 61-1 to 61-13 
Rosette, 29-13 
RotateLeft, Mathematica software, 73-13 
RotateRight, Mathematica software, 73-13 
Rotation group, representations, 59-9 to 59-10 
Rotation matrix, 65-5 
Rothblum, Uriel G., 9-1 to 9-23 
Rothblum index theorem, 26-8, 26-10 
Rounding error bounds, 37-14 
Rounding errors, 37-12, see also Error analysis 
Rounding mode, 37-12 
Round-robin tournament, 27-9 
Round-to-nearest standard, 37-12 
Routh-Hurwitz matrices 

stability, 19-3 

totally positive and negative matrices, 21-3 to 

21-4 

Routh-Hurwitz Stability Criterion, 19-4 
Row cyclic pivoting strategy, 42-18 
Row-cyclic pivoting strategy, 42-18 
Row-echelon form, 38—7 
Row echelon form (REF), 1—7 
RowReduce, Mathematica software 

linear systems, 73-20, 73-23 

matrix algebra, 73-10, 73-12 
Rows 

balanced signing, 33-5 

equivalence, 1-7, 23-5 

feasibility, 50-8 
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indices, 23-9 

matrices, 1-3 

pivoting, 46-5 

rank, 25-13 

row-major format, 74-2 

scaling, 9-20 

sign solvability, 33-5 

spaces, 2-6 

sum vectors, 27—7 

vectors, 1-3 
Row-stochastic matrices, 9-15 
Roy’s maximum root statistic, 53-13 
RRD, see Rank revealing decomposition (RRD) 
RREF, see Reduced row echelon form (RREF) 
rref command, Matlab software, 71-17 
RRQR (rank revealing QR) decomposition, 39-11 
Ruskeepaa, Heikki, 73-1 to 73-27 
Ryser/Nijenhius/Wilf (RNW) algorithm, 31-12 
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Sabinin algebra, 69-16 to 69-17 
Saddle point, 50-18 
Sadun, Lorenzo, 59-1 to 59-11 
Saiago, Carlos M., 34-1 to 34-15 
Sample canonical correlations and variates, 53-8 
Sample correlation coefficient, 52-9 
Sample covariance matrix, 53-8 
Sample mean, 53-4 
Sample points, 52-2 
Sample principal components, 53-5 
Samples, statistics and random variables, 52-2 
Sample spaces, 52-2 
Sampling, functional and discrete theories, 58-12 
Sandwich theorem, 28-10 
SAP (spectrally arbitrary pattern), 33-11 
Saturation digraphs, 25-6, 25-7 
Scalar matrix, 1-4 
Scalar multiple, vector spaces, 3—2 
Scalar multiplication 
matrices, 1-3 
vector spaces, 1—1 
Scalar transformation, 3—2 
Scaled sampling, 58-12 
Scaling 
doubly stochastic matrices, 27-10 
nonnegative matrices, 9-20 to 9-23 
Schatten-p norms, 17-5 
Schein rank, 25-13 
Schlaefli simplexes, 66-10, 66-11, 66-12 
Schneider, Barker and, studies, 26-3 
Schneider, Hans, 26—1 to 26-14 
Schneider’s theorem, 14—3 
Schoenberg characteristics, 66-8 
Schoenberg’s variation diminishing property, 
21-10 
Schoenberg transform, 35-10 
Schénhage, A., 47-8 
Schrédinger’s equation, 59-2, 59-6 to 59-9 
Schur algorithm, 64-8 


Schur complements 
bipartite graphs, 30-6 to 30-7 
determinants, 4—3, 4—4, 4-5 
inverse identities, 14-15 
partitioned matrices, 10—6 to 10-8 
random vectors, 52—4, 52—5 to 52-7 
symmetric indefinite matrices, 46-16 
Schur decomposition 
function computation methods, 11-11 
implicit restarting, 44-6 
pseudospectra, 16-3 


SchurDecomposition, Mathematica software, 73-19 


Schur-Horn theorem, 20-1 to 20-2 
Schur properties 
basis, 44-6 
form, 16-11 
inequalities, 14-2, 68-11 
linear prediction, 64-8 
product, 8-9 
relations, 68—4 
spectral estimation, 64—15 
Schur’s Lemma, 68—2 
Schur’s theorem 
eigenvalue problem, 43-2 
unitary similarity, 7-5 
Schur’s Triangularization theorem, 10-5 
Scilab’s Maxplus toolbox, 25-6 
Scores, estimation, 53-8 
Score vector, 27-9 
SCT, see Standard column tableau (SCT) 
SDP, see Semidefinite programming (SDP) 


Search engines, Markov chains, 54—4 to 54-5, see also 


Google (search engine) 

Seber, George A.F., 53-1 to 53-14 
Second canonical correlations and variates, 53—7 
Segment, Euclidean point space, 66-2 
Seidel matrix, 28-8 
Seidel switching 

graphs, 28-9 

matrix representations, 28-8 
Self-adjoints 

Hermitian matrices, 8—1 

linear operators, 5-5 

Schrédinger’s equation, 59-7 
Self-dual code, 61-3 
Self-inverse sign pattern, 33-3 
Self-polar cone, 51-5 
Semantic indexing, latent, 63-3 to 63-5 
Semiaffine characteristics, 65-2 
Semicolon, Maple software, 72-2 
Semiconvergence 

numerical methods, 54—12 

reducible matrices, 9-8, 9-11 
Semidefinite programming (SDP) 

applications, 51-9 to 51-11 

constraint qualification, 51-7 

duality, 51-5 to 51-7 

fundamentals, 51-1 to 51-3 

geometry, 51-5 

notation, 51-3 to 51-5 

optimality conditions, 51-5 to 51-7 
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primal-dual interior point algorithm, 51-8 to 51-9 linear prediction, 64—7 to 64-9 

results, 51-3 to 51-5 random signals, 64—4 to 64—7 

strong duality, 51-7 spectral estimation, 64-14 to 64-15 
Semidistinguished face, 26-8 Wiener filtering, 64-10 to 64-11 
Semimodules, 25-2 Signal subspace, 64-16 
Semipositive basis, 26-8 Signature 
Semipositive Jordan basis, 26-8 Hermitian forms, 12-8 
Semipositive Jordan chain, 26-8 symmetric bilinear forms, 12-3 
Semipositives Signature matrix 

fundamentals, 9-2 square case, 32-2 

Perron-Frobenius theorem, 26—2 Signature pattern, 33-2 
Semisimple algebras Signature similarity, 33-2 

general properties, 69-4, 69-5 Sign-central patterns, 33-17 

Lie algebras, 70-3 to 70-7 Sign changes, 21-9 
Semisimple eigenvalues, 4—6 Signed bigraph, 30-4 
Semistable matrices, 19-9 Signed bipartite graph, 30-1 
Semrl, Peter, 22-1 to 22-8 Signed 4-cockade, 30-4 
Sensitivity Signed digraphs, 33-2 

least squares solutions, 39-7 to 39-8 Signed singular value decomposition, 46-16 

linear programming, 50-17 to 50-18 Sign function, 11-12 
Separation Significand, 37-11 

alternative algebras, 69-10 Signing, 33-5 

eigenvalue problems, 15-2 Signless Laplacian matrix, 28-7 
Separation theorem, 25-11 Sign nonsingularity 
Separator, reordering effect, 40-16 rank revealing decomposition, 46-8 
Sesquilinear forms, 12-1, 12-6 to 12-7 sign-pattern matrices, 33-3 to 33-5 
Sets, nonnegative matrices, 9-23 Sign pattern, 30-4 
Setting up linear programs, 50-3 to 50-7 Sign pattern class 
Severin, Andrew, 60-13 complex sign and ray patterns, 33-14 
SGEEV, driver routine, 75-11 to 75-13 sign-pattern matrices, 33-1 
SGELS driver routine, 75-5 to 75-6 Sign-pattern matrices 
SGESVD, driver routine, 75—14 to 75-15 allowing properties, 33-9 to 33-11 
SGESV driver routine, 75-3 to 75-4 complex sign patterns, 33-14 to 33-15 
SGGEV, driver routine, 75-18 to 75-20 eigenvalue characterizations, 33-9 to 33-11 
SGGGLM, driver routine, 75-8 to 75-9 fundamentals, 33-1 to 33-3 
SGGLSE driver routine, 75-7 inertia, minimum rank, 33-11 to 33-12 
SGGSVD, driver routine, 75—22 to 75-23 inverses, 33-12 to 33-14 
Shader, Bryan L., 30-1 to 30-10 L-matrices, 33—5 to 33-7 
Shannon capacity, 28-9 orthogonality, 33-16 to 33-17 
Shannon’s Coding theorem, 61-3 to 61-4 powers, 33-15 to 33-16 
Shape, matrices, 1-3 ray patterns, 33-14 to 33-16 
Shapiro, Helene, 7-1 to 7-9 sign-central patterns, 33-17 
Sherman-Morrison, 14—15 sign nonsingularity, 33-3 to 33-5 
Shestakov, Ivan P., 69-1 to 69-25 sign solvability, 33-5 to 33-7 
Shift, symmetric matrix eigenvalue techniques, S-matrices, 33—5 to 33-7 

42-2 stability, 33-7 to 33-9 
Shift and invert spectral transformation mode, Sign potentially orthogonality (SPO), 33-16 
ARPACK, 76-7 Sign semistability, 33-7 

Shifted matrices, 42-2 Sign singularity 
Shifted QR iteration, 42-3 rank revealing decomposition, 46-8 
Shifts, polynomial restarting, 44-6 sign nonsingularity, 33-3 
Shor’s factorization algorithm Sign solvability, 33-5 to 33-7 

Grover’s search algorithm, 62-17 Sign stable, 33-7 

quantum computation, 62-6, 62-17 to 62-19 Sign symmetric, 19-3 
Show, Mathematica software, 73-5 Similarity 
Sign, P-6 change of basis, 3-4 
Signal model, 64-16 linear independence, span, and bases, 2-7 
Signal processing matrix similarities, 24—1 

adaptive filtering, 64-12 to 64-13 Similarity of matrix families 

arrival estimation direction, 64—15 to 64-18 classification I, 24—7 to 24-10 


fundamentals, 64—1 to 64-4 classification II, 24-10 to 24-11 
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fundamentals, 24—1 to 24-5 
property L, 24—6 to 24-7 
simultaneous similarity, 24—5 to 24-11 
Similarity-scaling, 9-20 
Simon’s problem, 62-13 to 62-15 
Simple algebras 
general properties, 69-4 
Lie algebras, 70-3 to 70-7 
Simple cycles 
matrix completion problems, 35-2 
sign-pattern matrices, 33-2 
Simple eigenvalues, 4—6 
Simple events, 52-2 
Simple graphs 
algebraic connectivity, 36-1 to 36-4, 36-9 to 36-10 
graphs, 28-1 
Simple linear regression, 52-8 
Simple row operations, 23-6 
Simple walk, 29-2 
Simplexes, 66-7 to 66-13 
Simplex method, 50-11 to 50-13 
Simplicial cones, 26—4 
simplify, Maple software, 72-8 
Simplify, Mathematica software 
eigenvalues, 73-15, 73-16 
fundamentals, 73-25 
matrix algebra, 73-12 
Simultaneous similarity 
classification I, 24—7 to 24-10 
classification II, 24-10 to 24-11 
fundamentals, 24—5 to 24-6 
Sin, Mathematica software, 73-26 
Sine, function computation methods, 11-11 
Single-input, single-output, time-invariant linear 
dynamical system, 49-14 
Single precision, 37-13 
Singleton bound 
convolutional codes, 61-12 
linear block codes, 61—5 
Singularity, isomorphism, 3—7 
Singular matrices, 1-12 
Singular pencils 
generalized eigenvalue problem, 43-2 
linear differential-algebraic equations, 55-7 
Singular-triplet, 15-6 
SingularValueDecomposition, Mathematica software 
decomposition, 73-18 
fundamentals, 73-27 
singular values, 73-17 
Singular value decomposition (SVD) 
accuracy, 46-2 to 46-5, 46—7 to 46-10 
algorithms, 45—4 to 45-12 
fundamentals, 5-10 to 5—12, 45-1 to 45-4 
LAPACK subroutine package, 75-13 to 75-15, 75-20 
to 75-23 
numerical stability and instability, 37-20 
orthogonal factorizations, 39-5 
SingularValueList, Mathematica software 
fundamentals, 73-27 
matrix algebra, 73-11 
singular values, 73-16 
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Singular values 
inequalities, 17—7 to 17-8, 17-9, 17-10 to 17-11 
Mathematica software, 73-16 to 73-18 
matrix equalities and inequalities, 14-8 to 14-10 
singular value decomposition, 5-10 
Singular values, high relative accuracy 
accurate SVD, 46-2 to 46-5, 46—7 to 46-10 
fundamentals, 46-1 to 46-2 
one-sided Jacobi SVD algorithm, 46-2 to 46-5 
positive definite matrices, 46-10 to 46-14 
preconditioned Jacobi SVD algorithm, 46-5 to 
46-7 
rank revealing decomposition, 46—7 to 46-10 
structured matrices, 46-7 to 46-10 
symmetric indefinite matrices, 46-14 to 46-16 
SingularValues, Maple software, 72-9 
SingularValues, Mathematica software, 73-27 
Singular values, problems 
generalized, 15-12 to 15-13 
perturbation theory, 15-6 to 15-7, 15-12 to 15-13 
relative perturbation theory, 15-15 to 15-16 
Singular values and singular value inequalities 
characterizations, 17—1 to 17-3 
eigenvalues, Hermitian matrices, 17—13 to 17-14 
fundamentals, 17—1 to 17-3 
generalizations, 17-14 to 17-15 
general matrices, 17-13 to 17-14 
inequalities, 17—7 to 17-12 
matrix approximation, 17-12 to 17-13 
results, 17-14 to 17-15 
special matrices, 17-3 to 17-5 
unitarily invariant norms, 17-5 to 17-7 
Singular value vector, 17-1 
Sinusoids in noise, 64-14 
Size, matrices, 1-3 
size command, Matlab software, 71-2 
eel condition number, 38-2 
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eel matrix condition number, 38-2 
ew-component, 56-11 
ew-Hermitian characteristics 
matrices, 1—4, 1-6 
spectral theory, 7-5, 7-8 
ew product flows, linear, 56-11 to 56-12 
ew-symmetric matrices 
direct sum decompositions, 2-5 
fundamentals, 1-4, 1-6 
invariance, 3—7 
kernel and range, 3-6 
ackness 
duality, 50-14, 51-6 
max-plus eigenproblem, 25-7 
optimality conditions, 51-6 
ack variables 
linear programming, 50-7 
linear programs, 50-8 
Slapnicar, Ivan, 42-1 to 42-22 
Slater’s Constraint Qualification, 51-7, 51-8 
Small oscillations, 59—4 
S-matrices 

sign-pattern matrices, 33-5 to 33-7 
SmithForm, Maple software, 72-16 
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Smith invariant factors 
rational canonical form, 6-13 
Smith normal form, 6-11 
Smith normal form 
canonical forms, 6—11 to 6-12 
matrix equivalence, 23-5 to 23-8, 23-6 
Smith normal matrix, 6-11 
Smooth curve, 61-10 
Smooth point, 24-8 
Soft information, 61-10 
Software, see also specific package 
freeware, 77—1 to 77-3 
pseudospectra computation, 16-12 
sol, Mathematica software, 73-21, 73-23 
Solution perturbation 
linear system perturbations, 38-2 
Solutions 
linear differential equations, 55-2 
matrix, inverse eigenvalue problems, 20-1 
systems of linear equations, 1-9 
Solution set, 1-9 
Solvability 
general properties, 69-5 
semisimple and simple algebras, 70-3 
Solvability index, 69-5 
Solvable radical algebras, 69-6 
Solve, Mathematica software 
eigenvalues, 73-14 
linear systems, 73-20, 73-21, 73-23 
Sorenson, D.C., 44-1 to 44-12, 76-1 to 76-10 
SOR (successive overrelaxation) methods, 41-3 to 
41-4 
Spacing 
Fourier analysis, 58-3 
functional and discrete theories, 58—12 
Span 
linear independence, 2—1 to 2-3 
span and linear independence, 2-1 
Spanning family, 25-2 
Spanning subgraphs, 28-2 
Spanning tree, 28-2 
Spans, max-plus algebra, 25-2 
Spare matrices 
fundamentals, 43-1 
Matlab software, 71—9 to 71-11 
SPARFUN directory, Matlab software, 71-10 
Sparity pattern, 46-8 
Sparse approximate inverse, 41-11 
SparseArray, Mathematica software, 73-6, 73-8, 73-9 
Sparse Cholesky factorization, 49-3 
Sparse direct solvers, 77-2 
Sparse eigenvalue solvers, 77—2 
Sparse iterative solvers, 77-3 
Sparse LU factorization, 49-3 
Sparse matrices 
analyzing fill, 40-10 to 40-13 
effect of reorderings, 40-14 to 40-18 
factorizations, 40—4 to 40-10 
fundamentals, 40-1 to 40-2 
Lanczos methods, 42-21 
large-scale matrix computations, 49-2 
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modeling, 40-10 to 40-13 
reordering effect, 40-14 to 40-18 
sparse matrices, 40-2 to 40-4 
unsymmetric matrix eigensvalue techniques, 43-9 to 
43-11 
Sparse matrix factorizations, 49-2 to 49-5 
Sparse nonsymmetric matrices 
modeling and analyzing fill, 40-11 
reordering effect, 40-15, 40-17 
Sparse symmetric positive definite matrices, 40-15 
Sparse triangular solve, 49-3 
Sparsity pattern, 9-21 
Sparsity structure, 40-4 
Special, Jordan algebra, 69-12 
Special boundary points, 18-3 to 18-4 
Special gate, 62-7 
Special linear group, 67-3 
Special matrices, Matlab software, 71-5 to 71-7 
Special-purpose indices, 63-9 
Specialty problem, 69-17 
Special unitary group, 67-6 
Spectra, nonnegative IEPs, 20-6 to 20-10 
Spectral absolute value, 17-1 
Spectral cones, 26-8 
Spectral Conjecture, 20-7 
Spectral density, 64-5 
Spectral estimation, 64-14 to 64-15 
Spectral factorization, 64—5 
Spectrally arbitrary pattern (SAP), 33-11 
Spectral norm 
matrix norms, 37-4 
unitarily invariant norms, 17-6 
unitary similarity, 7-2 
Spectral pair, 26-9 
Spectral projections, 55-8 
Spectral projector, 25-8 
Spectral properties, 21-8 
Spectral radius 
eigenvalues and eigenvectors, 4-6 
reducible matrices, 9-10 
Spectral Theorem 
Hermitian matrices, 8—2 
spectral theory, 7—5 to 7-6 
Spectral theory 
cone invariant departure, matrices, 26-8 to 26-10 
matrices, special properties, 7-5 to 7-9 
Spectral transformations 
ARPACK, 76-7, 76-8 
implicitly restarted Arnoldi method, 44-11 to 44-12 
Spectral value set, 16-12 
Spectrum 
adjacency matrix, 28-5 
eigenvalues and eigenvectors, 4-6 
numerical range, 18-3 to 18-4 
Spectrum localization, 14—5 to 14-8 
Spectrum of reducible matrices, 25-7 
Speed, methods comparison, 42-21 
Sphere-packing bound, 61-5 
Spin-factor, 69-13 
Split composition algebras, 69-8 
Split null extension, 69-6 
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Split quasi-associative algebras, 69-16 
Splitting theorems, 26-13 to 26-14 
spy command, Matlab software, 71-10 
Sqrt, Mathematica software, 73-17, 73-26 
Square case, 32-2 to 32-4 
Square complex matrix, 19-3 
Squared multiple correlation, 52-8 
Square linear system solution, 1-14 
Square matrices 
combinatorial matrix theory, 27-3 to 27-6 
fundamentals, 1-3, 1-4 
nonsingularity characteristics, 2-9 to 2-10 
stability, 19-3, 19-5, 19-9 
Square root, matrices, 11—4 to 11-5 
Squareroot-free method, 45-5 
SRRD, see Symmetric rank revealing decomposition 
(SRRD) 
SRT, see Standard row tableaux (SRT) 
SSYEV, driver routine, 75-10 to 75-11 
SSYGV, driver routine, 75—16 to 75-17 
Stability 
cone invariant departure, matrices, 26-13 to 26-14 
error analysis, 37-18 to 37-21 
group representations, 68-1 


linear differential-algebraic equations, 55-14 to 55-16 
linear ordinary differential equations, 55-10 to 55-14 


LTI systems, 57—7 
matrices, Maple software, 72-20 to 72-21 
matrix stability and inertia, 19-3 to 19-5 
pseudospectra, 16-2 
signal processing, 64—2 
sign pattern matrices, 33-7 
sign-pattern matrices, 33-7 to 33-9 
subspaces, linear differential equations, 56-3 
Stability and inertia 
additive D-stability, 19-7 to 19-8 
fundamentals, 19-1 to 19-2 
inertia, 19-2 to 19-3 
Lyapunov diagonal stability, 19-9 to 19-10 
multiplicative D-stability, 19-5 to 19-7 
stability, 19-3 to 19-5 
Staircase form, 57-9 
Standard basis, 2-3 
Standard column tableau (SCT), 50-13 
Standard deviations 
random vectors, 52-3 
statistics and random variables, 52—2 
Standard forms 
linear preserver problems, 22-2 to 22-4 
linear programming, 50-7, 50-7 to 50-8 
singular value decomposition, 45-1 
Standard inner product, 5-2, 13-23 
Standardized population principal component, 53-5 
Standard linear preserver problems, 22—4 to 22-7 
Standard map, 22-2 
Standard matrix, 3-3 
Standard row tableaux (SRT), 50-8 to 50-10 
Stars, multiplicity lists, 34-10 to 34-14 
Star-shaped sets, 20-6 
Starting vector, ARPACK, 76-6 
stat, Mathematica software, 73-15, 73-16 
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State 
classification, 54—7 to 54-9 
equation, control theory, 57-2 
estimation, control theory, 57-11 to 57-13 
feedback, 57-2, 57-7, 57-13 
observer, 57-12 
space, 54—1, 57-2 
stochastic and substochastic matrices, 9-15 
variables, 49-14 
vectors, 4-10, 57—2 
State-space dimension, 49-14 
State-space transformations 
frequency-domain analysis, 57-6 
LTI systems, 57—7 
Static feedback, 57-13 
Stationary characteristics, 64—4 to 65-5 
Stationary distribution 
Markov chain, 54—2 
stochastic and substochastic matrices, 9-15 
Statistical independence, 53-2 
Statistical inference, 53-12 to 53-13 
Statistics, see Probability and statistics applications 
Steady-state flux cone, 60-10 
Steady-state flux equation, 60-10 
Steady state vector, 4-10 
Stein studies, 26-14 
Stewart, Michael, 64—1 to 64-18 
Stewart studies, 44—4 
Stochastic and substochastic matrices, 9-15 to 9-17 
Stochastic hyperlink matrix, 63-11 
Stochastic spectral estimation, 64-14 
Stoichiometric coefficient, 60-10 
Stoichiometry matrix, 60-10 
Stopping criteria, 41-16 to 41-17 
Stopping criterion, ARPACK, 76-6 
Stopping matrices, 9-15 
Storage declaration, ARPACK, 76-5 to 76-6 
Strassen, V., 47-8 
Strassen’s algorithm, 47-3, 47-4 
Strassen’s formula, 47-3 
Strategies, matrix games, 50-18 
Stratification, 24-8 
Strengthened Landau inequalities, 27-9 
Strict column signing, 33-5 
Strict complementarity, 51-6 
Strict equivalence, pencils 
generalized eigenvalue problem, 43-2 
matrices over integral domains, 23-9 to 23-10 
Strictly block lower triangular matrices, 10-4 
Strictly block upper triangular matrices, 10-4 
Strictly copositive matrices, 35-11 to 35-12 
Strictly diagonally dominant matrices, 9-17 
Strictly similarity, 24-5 
Strictly unitarily equivalence, 43-2 
Strictly upper triangular matrices, 10-4 
Strict row signing, 33-5 
Strict signing, 33-5 
Strong Arnold Hypothesis, 28-9, 28-10 
Strong combinatorial invariants, 27-1, 27—3 to 
27-5 
Strong connections, 9-2 
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Strong duality 
duality and optimality conditions, 51-6 
semidefinite programming, 51-7 
Strongly connected components 
irreducible matrices, 29-7 
Jordan algebras, 69-13 
Strongly connected digraphs, 29-6 to 29-8 
Strongly inertia preserving, 19-9 
Strongly regular graphs, 28-3 
Strongly stable matrices, 19-7 
Strong nonsingularity, 47-9 
Strong Parter vertex, 34—2 
Strong preservation, 22-1 
Strong product, 28-2 
Strong rank, 25-13 
Strong sign nonsingularity, 33-3 
Strong stability, 37-18 
Structure and invariants, 27—1 to 27-3 
Structure constants, 69-2 
Structured matrices 
high relative accuracy, 46—7 to 46-10 
Maple software, 72-16 to 72-18 
Structured matrices, computations 
direct Toeplitz solvers, 48—4 to 48-5 
fundamentals, 48-1 to 48-4 
iterative Toeplitz solvers, 48-5 
linear systems, 48-5 to 48-8 
total least squares problems, 48-8 to 48-9 
Structured pseudospectrum, 16-12 
Structure index, 63-9 
Structure matrix, 27-7 
Stuart, Jeffrey L., 6-14, 29-1 to 29-13 
Studham, Matthew, 60-13 
Sturn-Liouville problem, 20-10 
Styan, Evelyn Mathason, 53-14 
Styan, George P.H., 52—1 to 52-15, 53-1 to 53-14 
Stykel, Tatjana, 55-1 to 55-16 
Subalgebra, 69-3 
Sub-bimodules, 69-6 
Subdigraphs, 29-2 
Subgraph, 28-2 
Submatrices 
fundamentals, 1-4, 1-6 
Gaussian and Gauss-Jordan elimination, 1-8 to 1-9 
inequalities, 17-7 
Matlab software, 71-1 to 71-3 
partitioned matrices, 10-1 to 10-3 
SubMatrix, Mathematica software, 73-13 
Submodules 
Bezout domains, 23-8 
modules, 70-7 
Submultiplicative properties, 18-6 
Subnormal floating point numbers, 37-11 
Suboptimal control problem, 57-15 
Subordinate matrix norms, 37—4 
Subpatterns, sign-pattern matrices, 33-2 
Subpermanents, 31—9 to 31-10 
Subrepresentation, 68-1 
Subroutine packages 
ARPACK, 76-1 to 76-10 
BLAS, 74-1 to 74-7 
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EIGS, 76-1 to 76-10 
LAPACK, 75-1 to 75-23 
subs command, Matlab software, 71-17, 71-18 
Subsemimodules, 25—2 
Subspaces 
direction, arrival estimation, 64-16 
direct sum decompositions, 2-5 
implicitly restarted Arnoldi method, 44-9 to 44-10 
iteration, 42-2 
nonassociative algebra, 69-3 
vector spaces, 1—2 
Substochastic matrices, 9-15 to 9-17 
Subtractive cancellation 
conditioning and condition numbers, 37-8 
floating point numbers, 37-15 
Subtractive cancellation, significant digits, 37-13 
Subtuple theorem, 20-7 
Successive overrelaxation (SOR) methods, 41-3 to 41-4 
Sufficient conditions, 20-8 to 20-10 
Sum 
characters, 68—5 
direct sum decompositions, 2-5 
vector spaces, 3-2 
sum command, Matlab software, 71-17 
Sum-norm, 37-2 
Sum of squares, residual, 52-8 
Sun lemma 
eigenvalue problems, 15-10 
singular value problems, 15-12 
Superposition Principle 
double generalized stars, 34-12 to 34-14 
mathematical physics, 59-1 
quantum computation, 62-1 to 62-2 
Sup-norm, 37-2 
supply, Mathematica software, 73-24 
Support 
linear inequalities and projections, 25-10 
scaling nonnegative matrices, 9-21 
square matrices, strong combinatorial invariants, 27-3 
Support line, 18-3 
surfc command, Matlab software, 71-15 
Surjective, kernel and range, 3-5 
Surplus variables, 50-7 
Suttle’s algebra, 69-15 
Suttle’s example, 69-8 
SVD, see Singular value decomposition (SVD) 
Sweedler notation, 69-20 
Sweep, Jacobi method, 42-18 
Switch, Mathematica software, 73-8 
Switching equivalent, 28-8 
Sylvester’s equation, 57-10, 57-11 
Sylvester’s Identity, 4-5 
Sylvester’s law of nullity, 14-13 
Sylvester’s laws of inertia 
congruence, 8-6 
Hermitian forms, 12-8 to 12-9 
symmetric bilinear forms, 12-4 
Sylvester’s observer equation, 57-12 
Sylvester’s theorem, 42-14 
Symbol curve, Toeplitz matrices, 16-6 
Symbolic mathematics, 71-17 to 71-19 
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Symbols, Toeplitz matrices, 16-6 
sym command, Matlab software, 71-17 
Symmetric algebra 
Lie algebras, 70-2 
tensor algebras, 13-22 
Symmetric matrices, see also Multiplicity lists 
direct sum decompositions, 2-5 
fundamentals, 1-6 
invariance, 3—7 
kernel and range, 3-6 
Maple software, 72-14 
semidefinite programming, 51-3 
Symmetric matrix eigenvalue techniques 
bisection method, 42—14 to 42-15 
comparison of methods, 42-21 to 42-22 
divide and conquer method, 42-12 to 42-14 
fundamentals, 42-1 to 42-2 
implicitly shifted QR method, 42-9 to 42-11 
inverse iteration, 42-14 to 42-15 
Jacobi method, 42-17 to 42-19 
Lanczos method, 42-19 to 42-21 
method comparison, 42-21 to 42-22 
methods, 42-2 to 42-5 
multiple relatively robust representations, 42-15 to 
42-17 
tridiagonalization, 42-5 to 42-9 
Symmetric properties 
asymmetric maps, 13-10 to 13-12 
bilinear forms, 12-3 to 12-5 
cone programming, 51-2 
definite eigenproblems, 75-15 to 75-17 
digraphs, 35-2 
dissimilarity, 53-13 
eigenvalue problems, 75-9 to 75-11 
factorizations, 38—15 to 38-17 
form, 12-1 to 12-5 
function, elementary, P-2 to P-3 
group representations, 68-10 to 68-11 
Hamiltonian, minimally chordal, 35-15 
Hermitian matrices, 8—1 
indefinite matrices, 46-14 to 46-16 
inertia set, 33-11 
Kronecker product, 51-3 
Lanczos process, 49-6 to 49-7 
maps, 13-10 to 13-12 
matrices, 1-4 
matrix games, 50-18 
maximal rank, 33-11 
minimal rank, 33-11 
positive definite matrices, 40-11 
product, 13-13 
reducible matrices, 9-11 to 9-12 
scaling, 9-20, 27-10 
tensors, 13-12 to 13-17 
Symmetric rank revealing decomposition (SRRD), 
46-14 
Symmetrization, 25-13 
Symmetrized rank, 25-13 
Sym multiplication, 13-17 to 13-19 
Symplectic group, 67—5 
syms command, Matlab software, 71-17 


Syndrome of y, 61-3 

Systematic encoder, 61-3 

Systems analysis, 58-7 

Systems of linear equations, 1-9 to 1-11 
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Table, Mathematica software 
linear programming, 73-24 
matrices, 73-6, 73-8 
singular values, 73-18 
vectors, 73—3, 73—4 
Tablespacing, Mathematica software, 73-7 
Take, Mathematica software 
matrices manipulation, 73-13 
vectors, 73—3 
TakeColumns, Mathematica software, 73-13 
TakeMatrix, Mathematica software, 73-13 
TakeRows, Mathematica software, 73-13 
Tam, Bit-Shun, 26-1 to 26-14 
Tam, T.Y., 68—1 to 68-11 
Tam-Schneider condition, 26-7 
Tangents, 24-1, 24-2 
Tangent space, 65-2 
Tanner, M., 61-11 
Tao, Knutson and, studies 
eigenvalues, 17-13 
Hermitian matrices, 8—4 
Taussky, Motzkin and, studies, 7-8 
Taylor coefficients, 49-15 
Taylor series, 37-20 to 37-21 
Taylor series expansion 
irreducible matrices, 9-5 
matrix function, 11-3 to 11-4 
Templates, ARPACK, 76-8 
Tensor algebras, 13-20 to 13-22, 70-2 
Tensor products, 10-8, 68-3 
Tensors 
algebras, 13-20 to 13-22 
decomposable tensors, 13-7 
Grassmann tensors, 13-12 to 13-17 
inner product spaces, 13-22 to 13-24 
linear maps, 13-8 to 13-10 
matrix similarities, 24-1 
multiplication, 13-17 to 13-19 
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products, 13-3 to 13-7, 13-8 to 13-10, 13-22 to 13-24 


symmetric tensors, 13-12 to 13-17 
Term-by-document matrix, 63-1 
Term rank 
combinatorial matrix theory, 27-2 
inertia, 33-11 
Term-wise singular value inequalities, 17-9 
Ternary Golay code, 61-8, 61-9 
Testing, 21-6 to 21-7 
Text, Mathematica software, 73-5 
TFQMR (transpose-free quasi-minimal residual) 
linear systems of equations, 49-14 
TGEVC LAPACK subroutine, 43-7 
TGSEN LAPACK subroutine, 43—7 
TGSNA LAPACK subroutine, 43—7 
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th cofactor, 4-1 

th compound matrix, 4-3 

th minor, 4-1 

Thompson’s Standard Additive inequalities, 17-8 


Thompson’s Standard Multiplicative inequalities, 17-8 


Thread, Mathematica software 
fundamentals, 73-26 
linear programming, 73-24 
linear systems, 73-20, 73-22 
Threshold pivoting, 38-10 
Ties-to-even standard, 37-12 
Tight sign-central matrices, 33-17 
Tikhonov regularization, 39-9 
Timed event graphs, 25—4 
Time-invariance, 57—2 
Time-map, 56-5 
Time space, 54-1 
Time varying linear differential equations, 56-11 
Tisseur, Higham and, studies, 16-12 
Tits system, 67-4 
Toeplitz-Block matrices, 48-3 
toeplitz function, Matlab software, 71-6 
Toeplitz IEPs (ToIEPs), 20-10 
Toeplitz-like matrices, 48-5 to 48-6 
Toeplitz matrices 
direct Toeplitz solvers, 48—4 to 48-5 
iterative Toeplitz solvers, 48-5 
least squares algorithms, 39-7 
linear prediction, 64-8 
Maple software, 72-18 
pseudospectra, 16—5 to 16-8 
structured matrices, 48-1, 48—4 
totally positive and negative matrices, 21-12 
Toeplitz operator, 16-5 
Toeplitz-plus-band matrices, 48-5, 48—7 to 48-8 
Toeplitz-plus-Hankel matrices, 48-5, 48-6 to 48-7 
ToIEPs (‘Toeplitz IEPs), 20-10 
Tolerance, Mathematica software 
matrix algebra, 73-11 
singular values, 73-17 
Top-down algorithm, 40-17 
Topic drift, 63-13 to 63-14 
Topological conjugacy, 56-5 
Topological equivalence, 56-5 
Torus, 70—4 
Total, Mathematica software 
fundamentals, 73-27 
linear programming, 73-24 
linear systems, 73-23 
matrices, 73-7, 73-9 
vectors, 73-3, 73-5 
Total degree, 23-2 
Total least squares problems, 39-2, 48-8 to 48-9 
Totally hyperacute simplexes, 66-10 
Totally nonnegative matrix, 46-10 
Totally positive matrices, 21-12 
Totally unimodular, 46-8 
Total memory, 61-12 
Total positive and total negative matrices 
deeper properties, 21-9 to 21-12 
factorizations, 21-5 to 21-6 
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fundamentals, 21-1 
properties, 21-2 to 21-4 
recognition, 21-6 to 21-7 
spectral properties, 21-8 
testing, 21-6 to 21-7 
Total signed compound (TSC) 
rank revealing decomposition, 46-8 
rank revealing decompositions, 46-9, 46-10 
Total support, 27-3 
Total variance, 53-5 
Tournament matrices, 27—8 to 27-10 
Tr, Mathematica software, 73-7 
Trace, 1-4, 3-3 
Trace, composition algebras, 69-8 
trace command, Matlab software, 71-17 
Trace debugging capability, ARPACK, 76-7 
Trace-minimal graph, 32-9 
Trace norm, 17-6 
Trace-sequence, 32-9 
Trailing diagonal, 15-12 
Trajectory, see Orbit 
Transfer function 
dimension reduction, 49-14 
frequency-domain analysis, 57—5 
signal processing, 64—2 
Transform, ATLAST, 71-22 
Transformations, linear, 3—1 to 3-9 
Transform principal component, 26-12 
Transience, 9-8, 9-11 
Transient class matrices, 9-15 
Transient state, 54—7 to 54-9 
Transient substochastic matrices, 9-15 
Transition graphs, 54-5 
Transition matrix 
coordinates and change of basis, 2-10 
Markov chains, 4—10, 54-1 
Transition probability, 4-10 
Transitive tournament matrices, 27—9 
Transpose 
linear functionals and annihilator, 3-8 
matrices, 1—4 
Transpose, Maple software, 72-3, 72-5, 72-9 
Transpose, Mathematica software 
eigenvalues, 73-15, 73-16 
fundamentals, 73-27 
linear systems, 73-23 
matrices manipulation, 73-13 
matrix algebra, 73-9 
Transpose-free quasi-minimal residual (TFQMR) 
linear systems of equations, 49-14 
Transvection, 67—3 
Tree of legs, simplexes, 66-10 
Trees 
algebraic connectivity, 36—4 to 36-6 
digraphs, 29-2 
graphs, 28-2 
multiplicity lists, 34-8 to 34-10 
sign pattern, 33-9 
vines, 34-15 
TREVC LAPACK subroutine, 43-6 
TREXC LAPACK subroutine, 43-7 
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Triangle, points, 65-2 
Triangle inequality 

inner product spaces, 5-2 

matrix norms, 37-4 

vector norms, 37—2 

vector seminorms, 37—3 
Triangular back substitution, 37-20 
Triangular factorization, 1-13 
Triangular linear systems, 38-5 to 38-7 
Triangular matrices, 10-4 
Triangular property, 35-2 
Tridiagonalization, 42-5 to 42-9 
Tridiagonal matrices, 21-4 
TridiagonalMatrix, Mathematica software, 73-6 
TridiagonalSolve, Mathematica software, 73-20 
Trigonometric form, 58-3 
Trilinear aggregating technique, 47-7 
Trilinear maps, 13-1 
Trinomial distribution, 52—4 
Trivial face, 26-2 
Trivial factors, 23-5 
Trivial linear combination, 2—1 
Trivial perfect codes, 61-9 
Trivial representation, 68-3 
Tropical semiring, 25-1 
TRSEN LAPACK subroutine, 43-7 
TRSNA LAPACK subroutine, 43—7 
Truncated singular value decomposition, 39-5 
Truncated Taylor series, 37-20 to 37-21 
Truncation errors, 37-12 
Tsatsomeros, Michael, 14—1 to 14-17 
TSC (total signed compound), 46-8 
Turbo codes, 61-11 
Turing machine, 62-2 
Turnpike theorem, 25-9 
Twisted factorization, 42-17 
Two-bit Controlled-U gate, 62-4 to 62-5 
Two-bit gate, 62-7 
Two(2)-design, 32-2 
Two-dimensional column-major format, 74-1 to 

74-2 

Two(2)-norm, 37—2 
Two-sided Lanczos algorithm, 41-7 
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FD (unique factorization domain), 23-2 
LV decomposition, 39-12 
nbounded region, 50-1 
neertainty, 59-7 
ncontrollable modes, 57—8 
ncorrelated vectors 

data matrix, 53-2 

random vectors, 52—4 
nderflow, 37-11 to 37-12 
ndirected graphs 

digraphs, 29-2 

modeling and analyzing fill, 40-10 
nicyclic graphs, 36-3 
niform distribution, 52—2 
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nimodular properties, 23-5 
nion, graphs, 28-2 
nipotent, linear group of degree, 67-1 
nique factorization domain (UFD), 23-2 
nique inertia, 33-11 
nique normalization, 23-3 to 23-4 
nital characteristics, 69-2 
nital hull, 69-5 
nital matrices mappings, 18-11 
nitary matrices 
adjoint operators, 5-6 
orthogonality, 5-3 
pseudo-inverse, 5-12 
singular value decomposition, 5-10 
Unitary properties, 5-2 
classical groups, 67-5 
equivalence, 7-2 
groups, 67-5 
Hessenberg matrix, 64-15 
invariance, 17—2, 18-6 
invariant norms, 17—5 to 17—7 
linear operators, 5-5 
Schrédinger’s equation, 59-7 
similarity, 7-2 
Unitary similarity 
invariant, numerical radius, 18-6 
matrices, special properties, 7—1 to 7-5 
transformation to upper Hessenberg form, 
43-4 
upper Hessenberg form, 43-4 
nit displacement rank matrix, 46-9 
nit round, 37-12 
nits, certain integral domains, 23-2 
nit triangular, 1-4 
nit vectors, 5-1 
nivariate linear model, 53-11 
niversal enveloping algebra, 70-2 
niversal factorization property, 13-3 to 13-4 
niversal property 
Lie algebras, 70-2, 70-3 
symmetric and Grassmann tensors, 13—14 to 
13-15 
niversal quantum gates, 62-7 to 62-8, see also 
Quantum computation 
nknown vector, 1-9 
nobservable modes, 57-8 
nordered multiplicities, 34-1 
nreduced Hessenberg matrix, 44-3 
nreduced upper Hessenberg, 43-3 
nsigned vectors, 33-5 
nstability 
linear differential-algebraic equations, 55-14 
linear ordinary differential equations, 55-10 
subspaces, 56-3 
Unsymmetric matrix eigensvalue techniques 
dense matrix techniques, 43-3 to 43-9 
fundamentals, 43-1 
generalized eigenvalue problem, 43-1 to 43-3 
sparse matrix techniques, 43-9 to 43-11 
Updating, least squares solutions, 39-8 to 39-9 
Upper Collatz-Wielandt numbers, 26-4 


Ct AGH Ct VIEL CG -Glnie! 


GS Ghrelin Gey .e} 


a 


SHS Ss mome mcs 


1-53 


1-54 


UpperDiagonalMatrix, Mathematica software, 
73-6 
Upper Hessenberg matrices 
Arnoldi factorization, 44-3 
block diagonal and triangular matrices, 10—4 
dense matrices, 43-3 
form, 43-3, 43-4 to 43-5 
implicit restarting, 44-6 
Krylov space methods, 41-8 
linear systems of equations, 49-13 
pseudospectra, 16-3 
pseudospectra computation, 16-11 
spectral estimation, 64-15 
Upper triangular properties 
block diagonal and triangular matrices, 10-4 
generalized eigenvalue problem, 43-2 
linear matrix, 38—5 
matrices, 1-4 
Upward eigenvalues, 34-11 
Upward multiplicity, 34-11 
URV decomposition, 39-11 
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Valency, graphs, 28-2 
Valuation, 36—7 
Value, matrix games, 50-18 
Vandermonde Determinant, 4—3 
Vandermonde matrices 
factorizations, 21-6 
linear systems conditioning, 37-11 
Maple software, 72-18 
rank revealing decomposition, 46-9 
structured matrices, 48-2 
symmetric indefinite matrices, 46-16 
totally positive and negative matrices, 21-3 
Variables 
pivoting, 50-10 
systems of linear equations, 1-9 
Variance 
principal component analysis, 53-5 
statistics and random variables, 52-2 
Variance-covariance matrix, 52-3 
Variety, simultaneous similarity, 24-8 
vars, Mathematica software 
linear programming, 73-24 
linear systems, 73-20, 73-21 
Vaserstein, Leonid N., 50-1 to 50-24 
Vec-function, 10-8 
Vector, Maple software, 72-1, 72-2 to 72-3 
vector, Maple software, 72-1 
Vector, Mathematica software, 73-3 
Vector generation, Maple software, 72-2 to 72-3 
Vector-Matrix products, Maple software, 72-6 
VectorNorm, Mathematica software, 73-27 
VectorQ, Mathematica software, 73—4 
Vectors, see also specific type 
balanced, 33-5 
control theory, 57-2 
Euclidean point space, 66-1 
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fundamentals, 1-1 to 1-3, 2-3 to 2-4, 3-2 to 
3-3 
Gauss elimination, 38—7 
Google’s PageRank, 63-11 
Maple software, 72-2 to 72-4 
Mathematica software, 73-3 to 73-5 
max-plus algebra, 25-1 
multiply, spare matrices, 43-10 
NMR protein structure determination, 60-2 
norms, error analysis, 37-2 to 37-3 
Perron-Frobenius theorem, 26—2 
query, 63-2 
seminorms, error analysis, 37-3 to 37-4 
sign solvability, 33-5 
space over, 1-1 
spaces, 1-2 
vector space method, 63-2 
Vectors, Maple software, 72-9 
Vector spaces 
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